Bandits with adversarial scaling

2020·arXiv

Abstract

Abstract

We study adversarial scaling, a multi-armed bandit model where rewards have a stochastic and an adversarial component. Our model captures display advertising where the click-through-rate can be decomposed to a (fixed across time) arm-quality component and a non-stochastic user-relevance component (fixed across arms). Despite the relative stochasticity of our model, we demonstrate two settings where most bandit algorithms suffer. On the positive side, we show that two algorithms, one from the action elimination and one from the mirror descent family are adaptive enough to be robust to adversarial scaling. Our results shed light on the robustness of adaptive parameter selection in stochastic bandits, which may be of independent interest.

1 Introduction

The multi-armed bandit setting is the cleanest paradigm to capture the tension between exploring information about the underlying system and exploiting the most profitable actions based on the current information. A decision-maker (or learner) repeatedly selects among a set of k actions also referred to as arms, earns the reward of the selected arm, and obtains feedback only about it. This creates a direct trade-off between learning about the performance of underexplored actions and earning the reward from those that seem the most profitable. This trade-off is prominent in applications such as display advertising where a platform needs to repeatedly select which ad to show in response to a particular pageview. The arms therefore correspond to the competing ads and, upon being shown, each ad can result in a click or not. We use this as a running example and assume that the goal of the learner is to maximize the total number of clicks (in practice, there are other goals such as revenue and user-experience, but we ignore those for simplicity).

Modeling this problem, one soon realizes that the two classical multi-armed bandit approaches fail to capture the essence of the setting. The two main approaches in the multi-armed bandit learning literature assume that the rewards obtained from each pageview are either completely adversarial or coming from identical and independent distributions (i.i.d.). However, clearly some ads have better quality and are consistently more clickable than others, so assuming a fully adversarial model seems to ignore a lot of useful structure in the data. On the other extreme, stochastic models assume that each ad has a fixed probability of being clicked whenever displayed (this is typically referred to as click-through-rate). In practice, click-through-rates are known to vary due to various factors: time of the day, day of the week, seasonalities (e.g. users tend to click more before Christmas and on Black Friday) but those factors affect ads uniformly. Tracà and Rudin [TR15] propose a model where the mean reward of an arm a at time t is a product

of the intrinsic quality of the arm and a seasonality term . In the display ads example, is the clickiness of the user behind her page-view and is the intrinsic quality of the ad. The motivation of [TR15] comes from retail: is the effect certain action has in the store (the product price, which items are on sale...) and is the number of customers in the store in that day. In their model, the seasonality effect is known to the algorithm.

We depart from their model by assuming we do not have access to . In display ads, the clickiness of users can be a function of a large number of covariates and it affects users in complex ways: certain users click more during the evening while other users click more during the day. Clickness of users in China is affected by Chinese New Year while clickiness of users in the US is affected by Thanksgiving. Estimating all those patterns is costly so it is useful to have models that exploit the structure of without having to estimate the seasonality/clickness model directly.

To obtain more robust algorithms we assume is adversarially chosen and hence the name adversarial scaling. If we think of as a seasonality effect, there is not an adversary per se in the motivation, but rather a very complex pattern which we choose not to assume anything about to get robust algorithms. However, can also model some actually adversarial effects. For example, an advertiser may create a botnet that does not click anything in the initial time, i.e. , aiming to prolong the algorithm’s exploration and hence her number of displays. This attack is very simple to run as it does not actively do anything and causes a cold start to the algorithm as the latter needs to appropriately disregard these insignificant samples.

As we demonstrate in this paper, despite the inherent stochasticity in these settings (best arm is better at each single round), we show that most stochastic algorithms perform poorly. This naturally generates the following question:

Our contribution. Tackling this question, we demonstrate that adaptive parameter selection is essential for robustness to adversarial scaling. In particular, we show a modification of the classical Action Elimination algorithm [EMM06] which we term AAEAS that uses the reward of the algorithm as a proxy for the number of rounds that matter, i.e. the total scaling . This allows it to adapt to the intrinsic qualities without being misled by the adversarial scaling in a way similar to the self-confident learning technique developed for small-loss bounds in adversarial bandits. The resulting guarantees hold with high probability which is an advantage of this method.

Aiming to understand whether this robustness is satisfied more broadly, we then focus on the Online Mirror Descent with Log-Barrier algorithm of Foster, Li, Lykouris, Sridharan, and Tardos [FLLInterestingly, when combined with the neat doubling trick of Wei and Luo [WL18], this algorithm (then termed BROAD) seamlessly adapts to the total scaling. Its resulting guarantees hold only for the weaker pseudo-regret notion but are stronger by a factor of k and the empirical performance is also enhanced.

To complement our study, we show two very simple attacks manage to make, to the best of our knowledge, all other stochastic algorithms perform ineffectively (detailed comparison in Section 4). Surprisingly these attacks work even against the recent breakthrough, Online Mirror Descent with Tsallis entropy [ZS19]. The latter has optimal regret guarantee for both stochastic, adversarial setting as well as regimes in between and is therefore considered as the best algorithm for such intermediate settings.

The first attack is a purely stochastic setting where all the arms have fixed but really small means where is the mean of the best action. This can be viewed as adversarial scaling with and . The bounds for these algorithms scales inversely to the absolute difference . In contrast, AAEAS and BROAD have boundes based on the normalized difference which provides great improvement when the means are small as typical in click-through-rates for display advertising.

The second attack is a cold start attack where in some initial period, the ; this can arise because either maliciously or organically as we discuss in the end of Section 4.2. The number of rounds that have passed becomes a rather irrelevant quantity in this case as what really matters is the . However, all algorithms other than AAEAS and BROAD use this number of rounds (many of them in ways that seem not fixable). The cold start therefore leads them to a really bad prior state and they take a very long time to recover. Surprisingly, the most effective stochastic algorithm Thompson Sampling [AG17] is prone to this attack even when the cold start is really short (see Figure 4).

Related work. Our paper lies in a broader line of work that tries to achieve enhanced guarantees when data exhibit a particular benign structure. The main structure that has been utilized is the data being i.i.d. across time where many algorithms achieve logarithmic guarantees at the existence of a large gap in . Under this structure, three previous lines of work study robustness of bandit learning to adversarial components, orthogonal to adversarial scaling. We briefly discuss them below and more elaborately review them in Section 4.

• Works on best of both worlds focus on the design of algorithms that achieve improved guarantees when the input is stochastic while retaining the regret guarantees of adversarial algorithms when this is not the case. This was introduced by Bubeck and Slivkins [BS12] and was further studied in a sequence of works [SS14, AC16, SL17, WL18, ZS19, ZLW19]. Interestingly, our work shows that the BROAD algorithm [WL18] is robust to adversarial scaling while Online Mirror Descent with Tsallis entropy [ZS19] that is considered superior fails at the presence of adversarial scaling. Our first algorithm can also be transformed to achieve such a best of both worlds guarantee if combined with the techniques of [BS12].

• Works on bandits with adversarial corruptions allow an adversary to corrupt some rounds. The adversary knows the distribution of the algorithm but not the randomness in the arm selection (at the extreme, this can capture the adversarial bandit problem). The goal is to a) achieve the improved guarantee when there is no corruption, b) have the performance gracefully degrade with the amount of corruption, while c) being agnostic to the amount of corruption. This was initially proposed by Lykouris, Mirrokni, and Paes Leme [LMPL18] who provided regret high-probability guarantees based on a multi-layering extension of active arm elimination. The guarantees were further improved by Gupta, Koren, and Talwar [GKT19] while further improvements were achieved for the weaker pseudo-regret notion [ZS19]. Our first algorithm can be made robust to adversarial corruptions if combined with the techniques of [LMPL18] while the second algorithm is also robust to adversarial corruptions for the notion of pseudo-regret.

• Last, a line of work aims to design attacks to stochastic algorithms when the adversary knows

the random arm selection of the algorithm at every round and not only the arm distribution [JLMZ18, LS19]. These works aim to minimize the amount of manipulation in the rewards needed for different objectives of the adversary and only provide attacks (instead of robust algorithms). Our algorithms obtain linear regret against such attacks (the same happens with all no-regret algorithms as these attacks give more power to the adversary than the adversarial bandit model).

Beyond stochasticity, many works exploit other benign properties in the data to enhance adversarial guarantees. Example properties include small variance of the losses [HK09, WL18], small effective loss range [CBS18], small variation in the losses across rounds [BLLW19], small loss of the best arm [AAGO06, Neu15, FLL, LST18, AZBL18], second-order excess loss [WL18], locally perturbed adversarial inputs [SS17] among others.

Finally, our model can be cast as a rank-1 assumption on the structure of the reward function. Rank-1 assumptions have been explored in various ways in the bandits literature, for example the rank-1 bandit model [KKS] and factored bandits [ZS18] assume that the action space have a cartesian product and hence the reward in each period can be represented as a matrix over which the rank-1 assumption is made. In other words, this rank-1 structure is on the space of actions unlike our model which is in the space of actions cross time. Another common setting where rank-1 assumptions ara made is in semi-parametric bandits [KWS18] where the loss function is a product of unknown parameters and features. A difference with respect to our model is that the features are observed by the learner while in our model the quality parameter is never observed by the learner.

2 Model

We study a multi-armed bandit setting with k arms where each arm is associated with an intrinsic mean parameter which is unknown to the learner. We define as the distribution of the reward of arm a in round t and we assume that it has positive measure only on the interval [0, 1]. The distributions are adaptively selected by the adversary subject to the constraint that the means must satisfy a rank-1 constraint, described below. Formally, the protocol between the learner and the adversary at each round t = 1..T is as follows:

• The learner chooses a distribution • The adversary chooses an adversarial quality parameter and distributions

For ease of presentation, we assume that the highest intrinsic mean is equal to This can be done by appropriately scaling down all the adversarial qualities replacing by

Regret. The goal of the learner is to maximize the aggregate reward she accumulates. To evaluate the performance of the learning algorithm, we compare this aggregate reward with the best strategy of the learner if she had access to the reward distributions of each arm – in that case, the optimal strategy is to select the arm with the highest intrinsic mean. The degradation that the algorithm incurs compared to this aware setting is captured by the notion of pseudo-regret in the purely stochastic setting. Even though we have an adversarial component on the rewards, the optimal arm to pull in each round is always an arm that maximizes irrespectively of the scaling chosen by the adversarial. This allows us to define pseudo-regret in the following way:

where . In other words, the pseudo-regret is worst-case over the sequence q(t) but it is in expectation over the draws

3 Algorithms robust to adversarial scaling

In this section, we show that two algorithms, one that we introduce from the active arm elimination family (Section 3.1) and one existing from the mirror descent family (Section 3.2) achieve the desired robustness to adversarial scaling. A common property of both algorithms is their adaptivity on the number of rounds; they do not scale with the number of rounds which is easily targeted by adversarial scaling attacks. In the next section, we show that other algorithms with improved guarantees in the stochastic regime that do not enjoy such adaptivity have their performance severely compromised at the presence of adversarial scaling.

3.1 Active Arm Elimination with Adversarial Scaling

The algorithm we introduce is based on Active Arm Elimination of Even-Dar, Mannor, and Mansour [EMM06], but is appropriately adapted to handle the adversarial scaling of the rewards.

Classical Active Arm Elimination keeps a set of active arms (initially all arms) and selects arms in a round-robin fashion among the active arms, updating the empirical mean of the selected arm as well as a confidence interval around it. The latter ensures that when samples are i.i.d. (same across rounds), with high probability, the actual mean lies within the confidence interval (this comes from an application of a Chernoff bound). Once the confidence intervals cease overlapping, we are confident that the dominated arm is not the arm with the highest intrinsic quality, thus it is safe to eliminate it from the active arms and never select it again.

Trying to extend this approach to adversarial scaling, one encounters some difficulties. One issue is that classic Active Arm Elimination selects arms deterministically. The adversary can easily cause linear regret to any deterministic policy by setting anytime the algorithm is about to pull the optimal arm. The natural way to get around this difficulty is to pull arms in the active set randomly. A more serious issue is that the number of samples (arm pulls) is no longer a meaningful quantity. The adversary may provide many samples initially with quality . These samples do not help in informing our estimates about where the intrinsic qualities lie; therefore treating these as real samples can provide a misleading picture of the confidence intervals. Ideally, we would like to use as effective samples the total adversarial quality at rounds where we selected each arm a, i.e. . Since we do not have access to this quantity, we need to design our confidence intervals in a way that will be robust to this adversarial scaling.

The main idea behind robustifying our confidence intervals is to use the reward of our algorithm as a proxy of the total effective samples of each arm. At the rounds that we selected arm , the reward of the algorithm is, in expectation, equal to the effective samples of (since ). Since we select each non-eliminated arm with equal probability, the total reward of the algorithm S serves as proxy for the effective rounds of each arm ( is a lower bound and S is an upper bound). Our algorithm, formalized in Algorithm 1, uses the proxy S instead of the unknown effective samples to construct confidence intervals for the arms; in fact, there is a single confidence interval CB(S).

Theorem 3.1. The AAEAS algorithm (Algorithm 1) run with has pseudo-regret at most:

The proof follows the standard active arm elimination analysis but replaces the samples of each arm by the proxy-created confidence interval. For completeness, we provide the Chernoff bound which we use in our analysis.

Lemma 3.2 (standard Chernoff bound). Let be independent random variables in

Proof of Theorem 3.1. To prove the guarantee, we need to ensure two properties: i) the arm highest intrinsic quality, with high probability, never gets eliminated and ii) we can bound the regret incurred by each suboptimal arm.

Our analysis is based on a few events occurring and we bound the failure probability of these events. We separate the time horizon in phases where a phase begins when exactly arms have been eliminated; denote by the round that the -th arm gets eliminated. First, the empirical reward R(a) that we experience for each arm is close to its expected reward , that is, for each arm a and each round t, with probability

This follows by a Chernoff bound (Lemma 3.2), since the rewards at round t are supported in [0, 1] and the arms are sampled uniformly at random from the set of active arms.

In a similar fashion, we can provide a similar confidence bound for the reward of our algorithm. For any round t, with probability

Setting all failure probabilities to , the probability that any of them fails is at most the remainder of the proof, we assume that none of these bounds fails.

First property. First, we establish that, when these bounds do not fail, the arm with the highest intrinsic mean does not become eliminated. Since the mean for any arm is at most the mean for , equation (1) implies:

We now show that, at every round, this difference is covered by the confidence bounds of our algorithm, i.e:

If then (3) holds as the second term of CB(S) is

. Otherwise, (2) and the optimality of

since the function is increasing for

Hence, (3) holds as the first term of CB(S) is . Since, in both cases, (3) holds, arm does not become eliminated.

Second property. Now we bound the regret coming from each suboptimal arm. Let’s consider the contribution to regret of arm . First we bound the difference of empirical rewards of two arms. By (1) and the fact that we have that:

We want to argue that once is large enough, the difference between the two arms CB(S), which leads to arm a getting eliminated and not contributing further regret. Let T(a) denote the time of the elimination of arm a. Since arm a is selected with equal probability with other active arms until then, the total expected regret from a is at most:

where denotes the value of at round T(a); recall that the latter is defined in (2). We will argue that arm a is eliminated by the time we have:

In order for an arm to be eliminated the difference needs to be relatively large. This difference can be expressed as:

The first inequality follows by applying (1) for both arm a and , and noting that . The second inequality is since . For the third inequality, first note that:

by the definitions of and , and since and for all . Combining (7) and (5), we therefore establish that . The third inequality then follows

directly since the function is increasing for

We can re-arrange the terms in equation (5) and obtain:

Combining (6) and (8) with the fact that , it holds:

The latter term is greater than the second term of the confidence bound CB(S). Comparing to the first, denoting the value of S at T(a), it holds that the latter RHS also dominates the first term of the confidence bound CB(S). As a result, if arm a was not eliminated by then, it gets eliminated when the expected reward of the algorithm becomes ; by (4) this implies that the expected contribution of arm a to the regret is at

most . Summing across all suboptimal arms and setting completes the proof.

Remark 3.3. While the proof is written in expectation over draws all arguments are high-probability arguments. The bound therefore can be converted to a high probability regret bound (as usual in stochastic bandits) with the difference that instead of capping the performance of each arm by as in the previous remark, we cap it by. Formally, we obtain that with probability we obtain the following bound on actual regret:

3.2 Online Mirror Descent with Log-Barrier

The second algorithm that we show to be robust to adversarial scaling lies in the mirror descent family and has a stronger regularizer, log-barrier. It was initially suggested by [FLL] for a fixed learning rate , who proved that it attains first-order bounds for pseudo-regret. The update on the probabilities is the following (Algotithm 3 in [FLL16] adapted to rewards):

where is such that p is a valid probability distribution.

Via using a neat doubling trick to update the learning rate , Wei and Luo [WL18] showed that, in fact, this algorithm can also attain stochastic guarantees (Algorithm 3 in [WL18] for the particular doubling trick). In particular, the algorithm halves the learning rate and restarts once:

where is the time of the last restart and is the importance sampling estimator. Their algorithm is a particular instantiation of a more general framework they termed BROAD.

We show that the BROAD algorithm (with no modification) is robust to adversarial scaling. The proof follows from replacing the potential the potential function in the proof of Theorem 10 in [WL18] to a potential that accommodates adversarial scaling. The remaining arguments are essentially the same as the ones used by [WL18] in their analysis of the stochastic setting.

Theorem 3.4. Online Mirror Descent with Log-Barrier with the above doubling (also known as BROAD) has pseudo-regret at most where is the minimum non-zero gap on intrinsic means.

We will use the following results proved in [WL18]. The first lemma corresponds to equation (27) in their paper and the second to equation (29). They are restated here for rewards instead of losses.

Lemma 3.5 ( [WL18]). There is a constant C such that for and in the BROAD algorithm it holds that:

Lemma 3.6 ([WL18]). Again in the context of BROAD, it holds that:

Proof of Theorem 3.4. The pseudo-regret of the algorithm can be expressed as:

where the inequality holds by noting that the difference in the means is at least . Note that in the adversarial scaling setting, we can bound the term in Lemma 3.6 by

By Lemma 3.5 as well as (11) and (10), setting as potential function holds that:

which leads to and concludes the proof.

Remark 3.7. Delving into Theorems 3.1 and 3.4, the reader may wonder whether the logarithmic dependence on T is necessary or whether it can be replaced by logarithmic dependence on at least for pseudo-regret guarantees. For BROAD, this is indeed the case with a simple change in the analysis: only appears inside the analysis to bound the divergence term and can be replaced by . For AAEAS, the logarithmic dependence on T appears in the confidence bound used by the algorithm. By adapting this confidence bound, we can obtain regret at most with probability leading to an expected regret of . If we have access to a known upper bound on Q, we can replace and obtain . At the absence of such an upper bound, it is not clear how to set the confidence bound in a way that achieves this goal.

4 Attacks against other stochastic algorithms

Besides AAE and BROAD which we previously discussed, there are few other algorithms available offering type of guarantees for stochastic bandits. In this sections we discuss how those perform in adversarial scaling settings. We describe two adversarial scaling attacks: (a) small means; and (b) cold-start. Besides AAEAS and BROAD we show that the remaining alternatives perform poorly in either of those cases. The algorithms we consider are:

• Upper Confidence Bound (UCB) [ACBF02] keeps track of the total reward r(a) of each arm and the number of times n(a) each arm was pulled. For each arm we compute the upper

confidence bound of each arm as and deterministically pull the arm with largest Ucb. As we previously mentioned, every algorithm that deterministically selects an arm can be easily fooled by adversarial scaling by setting when the algorithm is about to pull the optimal arm. We will see it is also tricked by much simpler (i.e. less adaptive) attacks.

• Thompson Sampling (TS) [AG17] is more easily described for the Bernoulli case where rewards are in {0, 1}. The algorithm keeps a prior for each arm a initially set with . In each round, the algorithm takes a sample from each prior, chooses the arm a with largest sampled value, observes the reward and updates the prior by increasing

• EXP3++ [SS14]: While traditional EXP3 algorithms of [ACBFS95] don’t offer guarantees in stochastic settings, this modification does by introducing an exploration parameter tuned for each arm as a function of its past comparative performance. Each arm is explored with probability given by such parameter and with remaining probability a standard EXP3 algorithm is run. This algorithm retains the EXP3 guarantees in the adversarial regime.

• Tsallis Entropy [ZS19]: A recent breatkthrough result provides an optimal algorithm (up to constants) for both stochastic and adversarial bandits via a standard mirror descent regularized by the Tsallis entropy. Remarkably, this algorithm requires no special tuning and no deviation from the standard mirror descent paradigm. The algorithm computes an unbiased estimator of the reward of each arm and then samples an arm from the probability distribution in the solution of the following maximization problem:

In Figure 1 we compare those algorithms in a purely stochastic instance with large means. As usually noted in the literature, the performance of Thompson Sampling is vastly superior than all other algorithms. On this instance, UCB, Tsallis and BROAD have similar perfomance, EXP3++ is somewhat worse followed by AAE and AAEAS which are notably worse. This is expected as they are the least adaptive. It is good to keep those in mind as we compare their performance on certain adversarial scaling scenarios.

Figure 1: Comparison of different bandit algorithms for a purely stochastic instance () with two arms with means . The cumulative in each round is the average of 100 runs of the each algorithm.

4.1 Small means

Even in the absence of adversarial scaling the AAEAS and BROAD algorithm start outperforming other algorithms as the means become smaller and smaller. Consider a purely stochastic instance (i.e. for all t) with only two arms with means and let . The traditional pseudo-regret bound obtained by stochastic bandits algorihtms is . The bound obtained by AAEAS and BROAD on the other hand is , which follows by viewing this purely stochastic problem as an instance with adversarial scaling with arms with and . In Figure 2 we compare the same algorithms in an instance with two Bernoulli arms having means 0.005 and 0.001. We see that while the performance of AAEAS and BROAD is unaffected by scaling the means down, the performance of UCB, AAE, Tsallis and EXP3++ degrades despite the fact that the relative strength of both arms remains almost the same.

Figure 2: A purely stochastic instance () with two arms with small means . The cumulative in each round is the average of 100 runs of the each algorithm.

A direct consequence of Theorem 3.1 and Theorem 3.4 is the following (we treat the number of arms k as a constant in order to unify the statements for AAEAD and BROAD):

Corollary 4.1. Given a stochastic bandit instance where rewards are supported in [0, 1] with means , then the performance of AAEAS and BROAD:

Interestingly, the performance of Thompson Sampling seems unaffected by adversarial scaling. It is not clear to us how to generalize the proof of [AG17] to explain the good performance of Thompson Sampling in the small mean regimes. We leave understanding this phenomenon as an open problem.

4.2 Cold start attack

A very effective type of attack against randomized algorithms (even Thompson Sampling) is the cold start attack, where for the first periods, the adversary chooses giving the algorithm the impression it has pulled a lot of arms although there are effectively no pulls. In Figure 3 we exhibit an extreme form of this attack, where we have a very long cold-start period () and from then on we have a standard stochastic instance with means 0.5 and 0.8. The performance of all algorithms except AAEAS and BROAD is severely hurt. In fact this performance degradation can become arbitrarly bad as . This happens for different reasons depending on the type of algorithm:

Figure 3: Comparison of different bandit algorithms on an extreme cold start instance: afterwards. Arm means are . We only plot the rounds after since the loss up to that point is zero.

the impression that the arms have much smaller mean than they actually have. Since the confidence bounds scale with the inverse square root of the empirical gap, the exploration phase can be arbitrarly extended as grows to infinity.

• Tsallis and EXP3++ are mirror-descent based algorithms that have a learning rate schedule that depends directly on the number of rounds (in either case). For those the cold-start will cause the learning rate to start at which is much smaller than the learning rate that would be required for a stochastic instance with larger means. Note that while BROAD is also based on mirror descent, the learning rate is adaptively tuned based on the rewards and it not directly depending on the numbers of rounds. This ability of adaptive tuning also enables first-order bounds for BROAD [FLL] and stems from the strong log-barrier regularizer that effectively deals with the variance in the second-order term. In contrast, Tsallis and EXP3++ have weaker regularizers and do not admit first-order bounds; for the same reason, we believe that it is unlikely that their learning rate can be appropriately tuned to circumvent this issue.

• Thompson Sampling is based on keeping a prior on the means of the arms. A large number of cold-start periods leads the algorithm to a state with very skewed priors. Once we reach the end of the cold-start period at each arm will have a beta distribution around instead of Beta(1, 1).

Figure 4: Comparing AAEAS with Thompson Sampling on a cold start instance () with means The y-axis is the cummulative loss of the algorithm averaged over 100 runs.

In Figure 4 we compare Thompson Sampling and AAEAS in an instance with a very small number of cold start rounds () and then run the algorithm for another T = 30000 rounds. Interestingly, we see that, even when the cold start is really small, the effect of the attack is long-lasting, which provides powerful evidence that the phenomenon described is, in fact, an actual concern when deploying Thompson Sampling. We also note that this phenomenon does not even need to be caused due to an adversarial source. It could well occur that the initial samples are less effective because, e.g., initially the advertised product is not yet well established which leads customers to prefer alternative options. In fact, recent work has suggested that, under competition, learning algorithms generally suffer from such an effect due to the exploration they need to perform in the beginning; see [MSW18] and [ALSW19] and for a relevant discussion.

4.3 Discussion on algorithms not run

Finally, we would like to mention that there are a few other algorithms with logarithmic stochastic guarantees that we decided to not include. The original best of both worlds algorithm SAO [BS12] and its follow-up SAPO [AC16] are rooted in AAE and switch to EXP3 if some test fails. Similarly, multi-layer AAE [LMPL18] provides a way to robsutify AAE to adversarial corruptions. Since the instances are mostly stochastic and therefore the tests are not expected to fail, their performance is strictly inferior to the one of AAE. Another line of work focuses on stochastic algorithms from the EXP3 family that can effectively select non-stationarities in the environment, e.g. the R.EXP3 algorithm [BGZ14]. These algorithms again come with horizon-dependent learning rates and are expected to suffer similarly with Tsallis and EXP3++ (that belong to the same family). Finally, for the same reason, we did not run the recent HYBRID algorithm [ZLW19] which extends upon Tsallis but again uses horizon-dependent learning rates.

5 Conclusion

In this work, we suggest a new intermediary model between stochastic and adversarial bandits where an adversary can rescale all rewards in a given round by the same factor. We show that two adaptive algorithms are robust to this adversarial scaling and provide two natural attacks that demonstrate that other stochastic algorithms are not.

There are two nice open questions coming from our work. AAEAS is weaker by a factor of k compared to BROAD; we believe that our analysis is tight in that matter but it would be interesting to see if some alternative modification of AAE can remove this dependence. Moreover, Thompson Sampling, although ineffective for the cold start attack, has very good performance in the small means attack; this suggests that its analysis could become more tight to scale with the ratio of the means (rather than their difference).

References

[AAGO06] Chamy Allenberg, Peter Auer, László Györfi, and György Ottucsák. Hannan consistency in on-line learning in case of unbounded losses under partial monitoring. In Algorithmic Learning Theory (ALT), 2006.

[AC16] Peter Auer and Chao-Kai Chiang. An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits. In Proceedings of the 29th Annual Conference on Learning Theory (COLT), 2016.

[ACBF02] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multi-armed bandit problem. Mach. Learn., 47(2-3):235–256, May 2002.

[ACBFS95] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Proceedings of IEEE 36th Annual Foundations of Computer Science, pages 322–331. IEEE, 1995.

[AG17] Shipra Agrawal and Navin Goyal. Near-optimal regret bounds for thompson sampling. J. ACM, 64(5):30:1–30:24, September 2017.

[ALSW19] Guy Aridor, Kevin Liu, Aleksandrs Slivkins, and Zhiwei Steven Wu. Competing bandits: The perils of exploration under competition. In 20th ACM Conference on Economics and Computation (EC), 2019.

[AZBL18] Zeyuan Allen-Zhu, Sébastien Bubeck, and Yuanzhi Li. Make the minority great again: First-order regret bound for contextual bandits. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.

[BGZ14] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. In Advances in Neural Information Processing Systems 27. 2014.

[BLLW19] Sébastien Bubeck, Yuanzhi Li, Haipeng Luo, and Chen-Yu Wei. Improved path-length regret bounds for bandits. arXiv preprint arXiv:1901.10604, 2019.

[BS12] Sébastien Bubeck and Aleksandrs Slivkins. The best of both worlds: Stochastic and adversarial bandits. In Proceedings of the 25th Annual Conference on Learning Theory (COLT), 2012.

[CBS18] Nicolò Cesa-Bianchi and Ohad Shamir. Bandit regret scaling with the effective loss range. In Algorithmic Learning Theory, pages 128–151, 2018.

[EMM06] Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of Machine Learning Research, 7:1079–1105, 2006.

[FLLDylan J Foster, Zhiyuan Li, Thodoris Lykouris, Karthik Sridharan, and Eva Tardos. Learning in games: Robustness of fast convergence. In Advances in Neural Information Processing Systems (NIPS), 2016.

[GKT19] Anupam Gupta, Tomer Koren, and Kunal Talwar. Better algorithms for stochastic bandits with adversarial corruptions. arXiv preprint arXiv:1902.08647, 2019.

[HK09] Elad Hazan and Satyen Kale. Better algorithms for benign bandits. In Proceedings of the 20th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2009.

[JLMZ18] Kwang-Sung Jun, Lihong Li, Yuzhe Ma, and Jerry Zhu. Adversarial attacks on stochastic bandits. In Advances in Neural Information Processing Systems, pages 3640–3649, 2018.

[KKSSumeet Katariya, Branislav Kveton, Csaba Szepesvari, Claire Vernade, and Zheng Wen. Stochastic rank-1 bandits. arXiv preprint arXiv:1608.03023, 2016.

[KWS18] Akshay Krishnamurthy, Zhiwei Steven Wu, and Vasilis Syrgkanis. Semiparametric contextual bandits. In International Conference on Machine Learning, pages 2781–2790, 2018.

[LMPL18] Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pages 114–122. ACM, 2018.

[LS19] Fang Liu and Ness B. Shroff. Data poisoning attacks on stochastic bandits. In International Conference on Machine Learning (ICML), volume abs/1905.06494, 2019.

[LST18] Thodoris Lykouris, Karthik Sridharan, and Éva Tardos. Small-loss bounds for online learning with partial information. In Proceedings of the 31st Annual Conference on Learning Theory (COLT), 2018.

[MSW18] Yishay Mansour, Aleksandrs Slivkins, and Zhiwei Steven Wu. Competing bandits: Learning under competition. In 9th Innovations in Theoretical Computer Science Conference (ITCS), 2018.

[Neu15] Gergely Neu. First-order regret bounds for combinatorial semi-bandits. In Conference on Learning Theory (COLT), 2015.

[SL17] Yevgeny Seldin and Gábor Lugosi. An improved parametrization and analysis of the EXP3++ algorithm for stochastic and adversarial bandits. In Proceedings of the 30th Conference on Learning Theory (COLT), 2017.

[SS14] Yevgeny Seldin and Aleksandrs Slivkins. One practical algorithm for both stochastic and adversarial bandits. In Proceedings of the 31th International Conference on Machine Learning (ICML), 2014.

[SS17] Ohad Shamir and Liran Szlak. Online learning with local permutations and delayed feedback. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.

[TR15] Stefano Tracà and Cynthia Rudin. Regulating greed over time. arXiv preprint arXiv:1505.05629, 2015.

[WL18] Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In Conference On Learning Theory, pages 1263–1291, 2018.

[ZLW19] Julian Zimmert, Haipeng Luo, and Chen-Yu Wei. Beating stochastic and adversarial semi-bandits optimally and simultaneously. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.

[ZS18] Julian Zimmert and Yevgeny Seldin. Factored bandits. In Advances in Neural Information Processing Systems, pages 2835–2844, 2018.

[ZS19] Julian Zimmert and Yevgeny Seldin. An optimal algorithm for stochastic and adversarial bandits. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pages 467– 475. PMLR, 16–18 Apr 2019.

Designed for Accessibility and to further Open Science