Object detection and classification have been widely adopted in autonomous systems, such as automated vehicles [1]–[3] and unmanned aerial vehicles [4], [5], as well as surveillance systems, e.g., smart home monitoring systems [6], [7]. These systems first perceive the surrounding environment via sensors (e.g., cameras, LiDARs, and motion sensors) that convert analog signals into digital data, then try to understand the environment using object detectors and classifiers (e.g., recognizing traffic signs or unauthorized persons), and finally make a decision on how to influence/interact with the environment (e.g., a vehicle may decelerate or a surveillance system raises an alarm).

While the cyber (digital) attack surface of such systems have been widely studied [8]–[11], vulnerabilities in the perception domain are less well-known, despite perception being the first and critical step in the decision-making pipeline. That is, if sensors can be compromised then false data can be injected and the decision making process will indubitably be harmed as the system is not acting on an accurate view of its environment. Recent work has demonstrated false data injection against sensors in a remote manner via either electromagnetic (radio frequency) interference [12], laser pulses (against microphones [13], or LiDARs [14]–[16]), and acoustic waves [17], [18]. These perception domain sensor attacks

Figure 1: A STOP sign image was injected into a camera by a projector, which was detected by YOLOv3 [21].

alter the data at the source, hence bypassing traditional digital defenses (such as crypto-based authentication or access control), and are subsequently much harder to defend against [19], [20]. These attacks can also be remote in that the attacker needn’t physically contact/access/modify devices or objects.

Among the aforementioned sensors, at least for automated systems in the transportation and surveillance domains, cameras are more common/crucial. Existing remote attacks against cameras are limited to, essentially, denial-of-service attacks [15], [22], [23], which are easily detectable (e.g., by tampering detection [24]) and for which effective mitigation strategies exist (e.g., by sensor fusion [25]). In this work, we consider attacks that cause camera-based image classification system to either misperceive actual objects or perceive non-existent objects by remotely injecting light-based interference into a camera, without blinding it. Formally, we consider creation attacks whereby a spurious object (e.g., a non-existent traffic sign, or obstacle) is seen to exist in the environment by a camera, and alteration attacks, in which an existing object in the camera view is changed into another attacker-determined object (e.g., changing a STOP sign to a YIELD sign or changing an intruder into a bicycle).

As it is not possible, due to optical principles, to directly project an image into a camera, we propose to exploit two common effects in optical imaging systems, viz., lens flare effects and exposure control to induce camera-based misperception. The former effect is due to the imperfection of lenses, which causes light beams to be refracted and reflected multiple times resulting in polygon-shape artifacts (a.k.a., ghosts) to appear in images [26], [27]. Since ghosts and their light sources typically appear at different locations, an attacker can overlap specially crafted ghosts with the target object’s without having the light source blocking it. Auto exposure control is a feature common to cameras that determines the amount of light incident on the imager and is used, for example, to make images look more natural. An attacker can leverage exposure control to make the background of an image darker and the ghosts brighter, so as to make the ghosts more prominent (i.e., noticeable to the detector/classifier) and thus increase attack success rates. Fig. 1 presents an example of a creation attack, where we used a projector to inject an image of a STOP sign in a ghost, which is detected and classified as a STOP sign by YOLOv3 [21], a state-of-the-art object detector.

Theoretically arbitrary patterns can be injected via ghosts. However, it is challenging to practically and precisely control the ghosts, in terms of their resolutions and positions in images, making arbitrary injection impracticable in some scenarios. Hence, we propose an empirical projector-camera channel model that predicts the resolution and color of injected ghost patterns, as well as the location of ghosts, for a given projector-camera arrangement. Experimental results show that at short distances attack success rates are as high as 100%, but at longer distances the rates decrease sharply; this is because at long distances ghost resolutions are low, resulting in patterns that cannot be recognized by the classifier.

To improve the efficacy of our attack, which we dub GhostImage, especially at lower resolutions, we assume that the attacker possesses knowledge about the image classification/detection algorithm. Based on this knowledge the attacker is able to formulate and solve an optimization problem to find optimal attack patterns, of varying resolutions, to project that will be recognized by the image classifier as the intended target class [28], [29]; i.e., the pattern projected will yield a classification result of the attacker’s choice. As the channel may distort the injected image (in terms of color, brightness, and noise), we extend our projector-camera model to include auto exposure control and color calibration and integrate the channel model into our optimization formulation. This results in a pattern generation approach that is resistant to channel effects and thus able to defeat a classifier under realistic conditions.

We use self-driving and surveillance systems as two illustrative examples to demonstrate the potential impact of GhostImage attacks. Proof-of-concept experiments were conducted with different cameras, image datasets, and environmental conditions. Results show that our attacks are able to achieve attack success rates as high as 100%, depending on the projector-camera distance. Our contributions are summarized as follows.

• We are the first to study remote perception attacks against camera-based classification systems, whereby the attacker induces misclassification of objects by injecting light, conveying adversarially generated patterns, into the camera.

• Our attack leverages optical effects/techniques, namely, lens flare and auto-exposure control, that are widespread and common, making the attack likely to be effective against most cameras. Furthermore, we incorporate these effects in

an end-to-end manner into an adversarial machine learningbased optimization framework to find the optimal patterns an attacker should inject to cause misperception.

• We demonstrate the efficacy of the attacks through experiments with varying image datasets, cameras, distances, and indoor to outdoor environments. Results show that GhostImage attacks are able to achieve attack success rates as high as 100%, depending on the projector-camera distance.

System and attack models are described, including two attack objectives and the attacker’s capabilities.

A. System Model

We assume an end-to-end camera-based object classification system (Fig. 2) in which a camera captures an image of a scene with objects of interest. The image is then fed to an object detector to crop out the areas of objects, and finally these areas are given to a neural network to classify the objects. Autonomous systems increasingly rely on such classification systems to make decisions and actions. If the classification result is incorrect (e.g., modified by an adversary), wrong actions could be taken. For example, in a surveillance system, if an intruder is not detected, the house may be broken-in without raising an alarm.

B. Threat Model

We consider two different attack objectives. In creation attacks the goal is to inject a spurious (i.e., non-existent) object into the scene and have it be recognized (classified) as though it were physically present. For alteration attacks an attacker injects adversarial patterns over an object of interest in the scene that causes the object to be misclassified.

There are two types of attackers with differing capabilities: Camera-aware attackers who possess knowledge of the victim’s camera (i.e., they do not know the configuration of the lens system, nor post-processing algorithms, but they can possess the same type of camera used in the target system), from which they can train a channel model using the camera as a black-box. With such capabilities, they are able to achieve creation attacks and alteration attacks. System-aware attackers not only possess the capabilities of the camera-aware attackers, but also know about the image classifier including its architecture and parameters, i.e., black-box attacks on the camera but white-box attacks on the classifier. With such capabilities, it is able to achieve creation attacks and alteration attacks as well, but with higher attack success rates.

Both types of attackers are remote (unlike [34]), i.e., they do not have access to the hardware or the firmware of the victim camera, nor to the images that the camera captures. We assume that both attackers are able to track and aim victim cameras [16], [22], [35].

In this section, we will introduce optical imaging principles, including flare/ghost effects and exposure control, which we

STOP STOP R-CNN CNN “There is a STOP sign”

Object Camera Digital Image Object Detection

E.g., [30]–[32] GhostImage E.g., [28], [33]

Figure 2: Camera-based object classification systems. GhostImage attacks target the perception domain, i.e., the camera.

will exploit to realize GhostImage attacks. Then, we will discuss the preliminaries about neural networks and adversarial examples that we will use to enhance GhostImage attacks.

A. Optical Imaging Principles

Due to the optical principles of camera-based imaging systems, it is not feasible to directly point a projector at a camera, hoping that the projected patterns can appear at the same location with the image of the targeted object, because the projector has to obscure the object in order to make the two images overlap. We prove this infeasibility in Appendix C. Instead, we exploit lens flare effects and auto exposure control to inject adversarial patterns.

Lens flare effects [27] refer to a phenomenon where one or more undesirable artifacts appear on an image because bright light get scattered or flared in a non-ideal lens system (Fig. 3). Ideally, all light beams should pass directly through the lens and reach the CMOS sensor. However, due to the quality of the lens elements, a small portion of light gets reflected several times within the lens system and then reaches the sensor, forming multiple polygons (called “ghosts”) on the image. The shape of polygons depends on the shape of the aperture. For example, if the aperture has six sides, there will be hexagon-shaped ghosts in the image. Normally ghosts are very weak and one cannot see them, but when a strong light source (such as the sun, a light bulb, a laser, or a projector) is present (unnecessarily captured by the CMOS sensor, though [36]), the ghost effects become visible. Fig. 3 shows only one reflection path, but there are many other paths and that is why there are usually multiple ghosts in an image.

Existing literature [27], [37], [38] about ghosts focused on the simulation of ghosts given the detailed lens configurations, in which the algorithms simulate every possible reflection path.

Figure 3: Ghost effect principle

Such white-box models are computationally expensive, and also requires white-box knowledge of internal lens configurations, thus are not suitable for our purposes. In Sections IV and V, we study flare effects in a black-box manner (more general than Vitoria et al. [39]), where we train a lightweight end-to-end model that is able to predict the locations of ghosts, estimate the resolutions within ghost areas, and also calibrate colors.

Exposure control mechanisms [40], [41] are often equipped in cameras to adjust brightness by changing the size of the aperture or the exposure time. In this work, we will model and exploit auto exposure control to manipulate the brightness balance between the targeted object and the injected attack patterns in ghosts.

B. Neural Nets and Adversarial Examples

We abstract a neural network as a function we omit the details of it due to the page limit. The input (width, height and RGB channels) is an image, is the output vector, and is the parameters of the network (which is fixed thus we omit it for convenience). A softmax layer is usually added to the end of a neural network to make sure that . The classification result is . Also, the inputs to the softmax layer are called logits and denoted as Z(x).

An adversarial example [28] is denoted as y, where y = x + is additive noise that has the same dimensionality with x. Given a benign image x and a target label t, an adversary wants to find a targeted attacks. Note that, in this paper, the magnitude of is not constrained below a small threshold, since the perceived images are usually not directly observed by human users. But we still try to minimize it because it represents the attack power and cost.

In this section, we will discuss how a camera-aware attacker is able to inject arbitrary patterns in the perceived image of the victim camera using projectors. We will discuss the possibilities of using other attack vectors in Section VII-B.

A. Technical Challenges

Since we assume that the attacker do not have access to the images that the targeted camera captures, he/she will have to be able to predict how ghosts might appear in the image. First, the locations of ghosts should be predicted given the relevant positions of the projector and the camera, so that the

Figure 4: Capture and projection are reverses of each other.

attacker can align the ghost with the image of the object of interest to achieve alteration attacks. Second, since a projector can inject shapes in ghost areas, the attacker needs to find out the maximum resolution of shapes that it can inject. Lastly, it is also challenging to realize the attacks derived from the position and resolution models above with a limited budget.

B. Ghost Pixel Coordinates

Given the pixel coordinates of the target object G (Fig. 4a), we need to derive the real-world coordinates of the projector so that we know where to place the projector in order to let one of the ghosts overlap with the image of the object. To do this, we derive the relationship between in two steps: We first calculate the pixel coordinates of the light source A given , and then we calculate G based on A.

Based on homogeneous coordinates [42], assuming the camera is at the origin of the coordinate system, we have

where M is the camera’s geometric model [42], a M can be trained from another (similar) camera, and then be applied to the victim camera. The coordinates of A is then , by the homogeneous transformation. Note that, A does not have to appear in the view of the camera, which makes the attack more stealthy (See [36] and Fig. 19 in the appendix).

In order to find the relationship of the pixel coordinates between light sources A and their ghosts G, we did a simple experiment where we moved around a flashlight in front of the camera [43], and recorded the pixel coordinates of the flashlight and the ghosts. Similar to [39], we observe that, for each G, we have (being constant), wherever A is (Fig. 5), and . This means the feasible region for the placement of the projector is large; to attack an autonomous vehicle, for example, it can be located on an overbridge, on a traffic island, or even in the preceding vehicle, or on a drone, etc. Finally, given and r,

With G’s coordinates, the attacker is able to predict the pixel location of ghosts and try adjusting the position and orientation (which implies the angle) of the light source in the real world so as to align one or more ghosts with the image of the object, whose pixel coordinates can be calculated using (1) similarly.

Figure 5: Ghost position v.s. light source position. Crosses are light source at different locations and the circles are the according biggest ghosts (as examples).

C. Ghost Resolution

In our daily life, ghosts normally appear as pieces of single-color polygon-shaped artifacts; this is because the light sources that cause these regular ghosts are single-point sources of light that have just one single color, such as light bulbs, flashlights, etc. In this work, however, we find out that one is able to bring patterns into these ghost areas by simply using a low-cost projector, a special source of light that shines variant patterns in variant colors. For example, in Fig. 1, an image of a STOP sign that is projected by a projector, appears in one of the ghost areas in the image; this is because the pixel resolution of the projector is high enough that multiple light beams in different colors (got reflected among lenses and then) go into the same ghost. In this subsection, we study the resolution of the patterns in ghost areas 1. Let us first define the throwing ratio of a projector. In Fig. 4b, let plane S be the projected screen (e.g. on a wall), whose height and width are denoted as h and w, respectively. The distance is called the throwing distance. The throwing ratio of this projection is . The (physical) size of the projected screen at the victim camera’s location is denoted , a part of which is captured by the CMOS sensor of the camera in the ghost area, and we denote the (physical) size of that area as . Let us also define the resolution of the entire projected screen as in terms of pixels (e.g., ), and the resolution of the ghost as . Clearly, there is a linear relationship among them: Finally, we can calculate the resolution of the ghost given d and :

Here, is a constant because the size of the lens is fixed; e.g., the camera [43] has cm.

D. Attack Realization and Experiment Setup

According to Eq. 3, if the attacker wants to carry out long-distance and high-resolution GhostImage attacks, it needs a

Figure 6: (Left) Attack setup diagram. (Middle) In-lab experiment setup. (Right) Attack equipments: We replaced the original lens of the NEC NP3150 Projector [44] with a Canon EFS 55-250 mm zoom lens [45].

projector with a large throwing ratio . However, the factory longest-throw lenses (NEC NP05ZL Zoom Lens [46]) of our projector can achieve a throwing ratio of maximum 7.2 (which means at one meter), and expensive (about $1600). Instead, we use a cheap ($80) zoom lens (Fig. 6, right) [45] that was originally designed for Canon cameras. In our experiments, such a configuration is interestingly feasible2 (Fig. 6), achieving the maximum throwing ratio of 20 when the focal length is 250 mm, which means that at a distance of one meter, resolution attacks can be achieved. See Sec. VII-A for more discussion on lens and projector selection.

Fig. 6 (left) shows a general diagram of GhostImage attacks, where the light source (i.e., a projector) is pointing at the camera from the side, so that the camera can still capture the object (e.g., a STOP sign) for alteration attacks. The light source injects light interference (marked in blue) into the camera, which gets reflected among the lenses of the camera, resulting in ghosts that overlap with the object in the image. Accordingly, a photo of our in-lab experiment setup is given in Fig. 6. The Canon lens was loaded in the NEC projector, though it cannot be seen in the photo. We will evaluate our attack on three different cameras (Sec. VI-B3).

To mount a creation attack, the attacker computes the maximum resolution for the ghost with a distance d based on (3), and then downsamples the target image to the resolution in order to fit in the ghost area. The attacker chooses downsampling as a heuristic approach because he/she is not aware of the classification algorithm. We present in Fig. 7 some examples of downsampling a STOP sign image.

To mount alteration attacks, in addition to (3) for downsampling, the attacker also needs to consider the pixel coordinates (Eq. 2) of the ghost because the attacker needs to align the ghost

2Because projectors and cameras are dual devices (Fig. 4), their lenses are interchangeable.

Figure 7: Downsampling examples. We actually upsampled these images for the sake of presentation, otherwise they were too tiny to show.

We substantiate camera-aware attacks on an image classifi-cation system that we envision would be used for automated vehicles. Specifically, images, taken by an Aptina MT9M034 camera [43], are fed to a traffic sign image classifier trained on the LISA dataset [48]. In Sec. VI, we will evaluate classification systems for other applications, with different cameras and different datasets.

1) Dataset and neural network architecture: In order to train an unbiased classifier, we selected eight traffic signs from the LISA dataset [48] (Table VI in Appendix A). The network architecture (Table V in Appendix A) is identical to [30]. We used 80% of samples from the balanced dataset to train the network and the rest 20% to test the network; it achieved an accuracy of 96%.

2) Evaluation methodology: The evaluation procedure for alteration attacks is detailed in Algorithm 1 in which we iterated five distances, m source classes, m target classes. For each target class, we sampled k images randomly from the dataset. For every combination, we first downsampled the target image based on (3), and projected the image at the camera using the NEC projector. We then took the captured image, cropped out the ghost area, and used the classifier to classify it. If the classification result is the target class, we count it as a successful

Algorithm 1: GhostImage Attack evaluation procedure (m: number of classes)

Figure 8: Camera-aware attack examples at one meter in perception domain. Left: Creating a Merge sign. Right: Altering a STOP sign (in the background) into a Merge sign.

attack. The procedure for creation attacks is slightly different: Rather than printed traffic signs, we placed a blackboard as the background as it helped us locate the ghosts. Given a throwing radio of 20 (thanks to the Canon lens) we evaluated five different distances from one meter to five meters. Based on (3), they resulted in , and resolutions, respectively.

3) Results: The results about attack success rates of camera-aware attacks at varying distances are shown in Table I (Fig. 8 illustrates two successful camera-aware attacks). For the digital domain, we simply added attack images on benign images x as . Based on these experiments, we observe: First, as the distance increases, the success rate decreases. This is because lower-resolution images are less well recognized by the classifier. Second, digital domain results are better than perception domain one, because images are distorted by the projector-camera channel effects. Third, creation attacks result in higher success rates than alteration attacks do because in alteration attacks there are benign images in the background, encouraging the classifier to make correct classifications. We will address these issues in the next section, so as to increase the overall attack success rate.

There are some limitations of the camera-aware attack intro- duced in the previous section. First, increasing distances results in lower success rates because the classifier cannot recognize the resulting low-resolution images. Second, there are large gaps between digital domain results and perception domain results, as channel effects (which cause the inconsistency between the intended pixels and the perceived pixels) are not taken into account. In this section, we resolve these limitations and improve GhostImage attacks’ success rates by proposing a framework which consists of a channel model that predicts

Table I: Camera-aware attack success rates

the pixels perceived by the camera, given the pixels as input to the projector, as well as an optimization formulation based on which the attacker can solve for optimal attack patterns that cause misclassification by the target classifier with high confidence.

A. Technical Challenges

First, the injected pixel values are often difficult to control as they exhibit randomness due to variability of the channel between the projector and the camera, thus the adversary is not able to manipulate each pixel deterministically. Second, to achieve optimal results, the attacker needs to precisely predict the projected and perceived pixels, thus channel effects must be modeled in an end-to-end manner, i.e., considering not only the physical channel (air propagation), but also the internal processes of the projector and the camera. Lastly, the resolution of attack patterns is limited by distances and projector lens (Eq. 3), thus the ghost patterns must be carefully designed to fit the resolution with few degrees of freedom.

B. System-aware Attack Overview

The system-aware attacker aims to find optimal patterns that can cause misclassification by the target classifier with high confidence by taking advantage of the non-robustness of the classifier [28]. We adopt an adversarial example-based optimization formulation into GhostImage attacks, in which the attacker tries to solve

where is the digital attack pattern as input to the projector, y is the perceived image of the object of interest under attacks, t is the target class, and represents the targeted neural network. -norm that measures the magnitude of a vector, and is a loss function indicating how (un)successful we aim to minimize the power of the projector required for a successful attack, meanwhile maximizing the successful chance of attacks. The relative importance of these two objectives is balanced by a constant c. In Sec. V-D, we will detail (4) in terms of how we handle being a non-negative random tensor that is also able to depict grid-style patterns in different resolutions.

More importantly, in (4) y is the final perceived image used as input to the classifier, which is estimated by our channel model in an end-to-end style (Fig. 9), in which is the input to the projector, and y is the resulting image captured by the camera. The model can be formulated as

where is the ghost model that estimates the perceived adversarial pixel values in the ghost. For simplicity we let because the attacker possesses same type of the camera so that x can be obtained a priori, and is the

Figure 9: Projector-camera channel model

auto exposure control that adjusts the brightness. Sec. V-C introduces the derivation of (5).

Next, we will first present the channel model, and then formulate the optimization problem for finding the optimal adversarial ghost patterns.

C. Projector-Camera Channel Model

where are constants derived from the data. is the maximum illuminance of the projector at a distance of one meter. Such a sigmoid-like

Figure 10: Illuminance depends on the RGB amplitude the light bulb intensity .

function captures the luminescence saturation property of the projector hardware.

b) How does the perceived x depend on I?: In the same experiments we also recorded the RGB value of the ghost (with a blackboard as background (in order to reduce ambient impacts), and a piece of white paper (x) that was also on the blackboard but did not overlay with the ghost. Their data are shown in Fig. 11, from which we can derive the dimming ratio that measures the change of exposure/brightness:

where is the ambient lighting condition in illuminance which differs from indoors to outdoors for instances. From this equation, we see that in an environment with static lighting condition, as the luminescence of the projector increases, the dimming ratio decreases, hence the objects become darker. With (7), the adversary is able to conduct real-time attacks by simply plugging in the momentary .

(the lower subplot of Fig. 11) depends on I in two ways:

On one hand, the last term I increases the intensity of ghosts, but on the other hand the dimming ratio dims down ghost, whereby is a trainable constant. With this, we can rewrite the perceived flare as

where is the color calibration matrix to deal with color distortion, which will be discussed in Section V-C2. The term normalizes . In the end, we have the channel model

Compared to (5),

Figure 11: Perceived RGB values v.s. illuminance.

With (8), the attacker is able to predict how bright and what colors/pixel values the ghost and the object will be, given the projected pixels, the power of the projector, and the distance.

2) Color calibration: Considering a dark background (i.e., x = 0), (8) can be simplified as , where is a matrix (as three color channels) that calibrates colors. Both y and are column vectors. should be an identity matrix for an ideal channel, but due to the color-imperfection of both the projector and the camera, needs to be learned from data. To simplify notations, we define corrected x and y as

We did another set of experiments where we collected n = 100 pairs of with dark background (to make x = 0), with being assigned randomly, and . We grouped them into X and Y :

where both matrices. We compute solving

This is known as a non-homogeneous least square problem [42], and it has a closed-form solution:

Plugging back to (8) completes our channel model.

3) Model validation: Fig. 12 demonstrates the accuracy of our channel model. In it the left image is the original input to the projector, the middle image is the estimated output from the camera based on our channel model (Eq. 8), and the image on the right is the actual image in a ghost captured by the camera. As can be seen, the difference between the actual and predicted is much less than the actual and original. While blurring effect

Figure 12: An example of channel model prediction

is apparent in the actual y, we do not model it but the success rates are still high despite it. As we will see in Section VI, our channel model is general enough that once trained on one camera in one environment, it can be transferred to different environments and different cameras without retraining.

D. Optimal Adversarial Projection Patterns

In long-distance, low-resolution GhostImage attacks there are only a few pixels in the ghost area. A camera-aware attacker’s strategy is to simply downsample attack images into low resolutions, but that does not result in high success rates. While (4) is abstract, for the rest of this subsection, we will progressively detail it and show how it can be solved in light of the channel model to improve attack success rates. We will start with the simplest case where adversarial perturbations are random noise (Sec. V-A). Then, single-color ghosts will be introduced. Later, we will consider how to find semi-positive additive noise due to the fact that superposition can only increase perceived light intensity but not decrease it. Finally, we examine the optimization problem to find optimal ghost patterns in grids at different resolutions.

1) Single-color ghost: Let us consider the simplest case first where the random noise is drawn from one single Gaussian distribution for all three channels, i.e., , where the size of is with w and h representing the width and height of the benign image x. This is because the values of each pixel that appear in the ghost area follow Gaussian distributions according to statistics obtained from our experiments. The adversary needs to find and such that when is added to the benign image x, the resulting image y will be classified as the target class t. That said, the logits value (Section III-B) of the target class should be as high as possible compared with the logits values of other classes [29]. Such a difference is measured by the loss function

where is the expectation of logits values at class i of input y. Term is the highest expected logits value among all the classes except the target class t, while is the expected logits value of t. Here, controls the logits gap between larger the is, the more confident that is successful. The attacker needs as low as possible so that the neural network would classify y as Class t. Most importantly, y is computed based on our channel model (Eq. 8), so that the optimizer finds the optimal ghost patterns that are resistant to the channel effects. Unfortunately, due to the complexity of neural networks, the expectations of logits values hard to be expressed analytically; we instead use Monte Carlo methods to approximate it:

where T is the number of trials, and is of the j-th trial.

Meanwhile, the adversary also needs to minimize the magnitude of to reduce the attack power and noticeability, as well as its peak energy consumption, quantified by expectation of the magnitude of is

Putting (9) and (10) together with a tunable constant c, we have our optimization problem for the simplest case

Here, is the lower bound of the standard deviation , meaning that the interference generator and the channel environment can provide random noise with the standard deviation of at least . When , the adversary is able to manipulate pixels deterministically. Therefore, when we fix as in the optimization problem, the attack success rate when deploying would be the lower bound of the attack success rate. In other words, the adversary equipped with an attack setup that can produce noise with a lower variance (than ) can carry out attacks with higher success rates. Therefore, we can simplify our formulation by removing the constraint about , so the optimization problem becomes

For the rest of the paper we will simply use to denote .

Since in (11) there is only one variable that the adversary is able to control, it is infeasible to launch a targeted attack with such few degrees of freedom. As a result, the adversary needs to manipulate each channel individually. That is, for each channel, there will be an independent distribution from which noise will be drawn. This is feasible because noise can appear in different colors in the ghost areas in which three channels are perturbed differently when using projectors. Let us decompose , where the dimension of , and they follow three independent Gaussian distributions

, σ, σ, σ.

Here, are the means and the standard deviations () of the three Gaussian distributions, respectively. The expectation of such is then

Figure 13: Biased penalty

(10) is a special case of (12) when . We denote . Hence, similar to (11), we have the optimization problem for single-color perturbation

where

Here is to center the global minimum at being zero, and substracting is to lower the minimum to be zero but it does not change the optimization results so we will omit it. An instance of (14) when is plotted in Fig. 13 in comparison with the norm. With the same absolute value, while the norm treats positive perturbation and negative perturbation equally, the biased penalty function punishes the negative values more than the positive one, encouraging the optimization algorithm to find positive . We adopt it into our optimization formulation

and in the experiments we set and .

3) Ghost grids: Since projector’s pixels are arranged in grids, the attack patterns are in grids as well, especially in lower resolutions. We enable with patterns in different resolutions. Such a grid pattern can be composed of several blocks i.e., where is the number of rows, columns, and channels of a grid pattern, respectively, in terms of blocks. In a word, is the perturbation block at i-th row, j-th column and k-th channel. A block is a random matrix and its size is , so that the size of

Figure 14: A grid pattern when and is a three-dimensional matrix. (b) The resulting perturbation pattern.

the elements in the random matrix is i.i.d. drawn from a Gaussian distribution, i.e., .

The adversary can find the optimal grid pattern by solving the optimization problem as in (15) in which

where is the mean of the block . See Fig. 14a for an illustration of the dimensionality of , and Fig. 14b for the resulting pattern in color.

In this section, we consider camera-based image classifica- tion systems, as used in self-driving vehicles and surveillance systems, to illustrate the potential impact of our attacks. We present proof-of-concept system-aware attacks in terms of attack effectiveness, namely how well system-aware attacks perform in the same setup as camera-aware attacks (Section IV-E), and attack robustness, namely how well system-aware attacks are when being evaluated in different setups.

We will again use attack success rates (Algorithm 1) as our metric. We used the Adam Optimizer [50] to solve our optimization problems. There are two sets of results: Emulation results refer to the classification results on emulated, combined images of benign images and attack patterns using our channel model (Equation 8). Emulation helps us conduct scalable and fast evaluations of GhostImage attacks before conducting real-world experiments4. Experimental results refer to the classification results on the images that are actually captured by the victim cameras when the projector is on.

A. Attack Effectiveness

To compare with camera-aware attacks, system-aware attacks are evaluated in a similar procedure (Algorithm 1), targeting a camera-based object classification system with the LISA dataset

Figure 15: System-aware creation and alteration

and its classifier, and the Aptina MT9M034 camera [43], in an in-lab environment.

1) Creation attacks: For emulated creation attacks, all distances (or all resolutions) yield attack success rates of 100% (Fig. 15), which means that our optimization problem is easy to solve. In terms of computational overhead, we need roughly 30 s per image at -resolution, and above (because of more degrees of freedom) using an NVIDIA Tesla P100 [51]. Fig. 16a shows examples of emulated attack patterns for creation attacks, along with the images of real signs on the top. Interestingly, high-resolution shapes do look like real signs. For example, we can see two vertical bars for ADDEDLANE, and also we can see a circle at the middle south for STOPAHEAD, etc. These results are consistent with the ones from the MNIST dataset [52] where we could also roughly observe the shapes of digits. Secondly, they are blue tinted because our channel model suggests that ghosts tend to be blue, thus the optimizer is trying to find “blue” attack patterns that are able to deceive the classifier.

Interestingly, the all k resulting patterns of solving the optimization problem targeting one class from k different (random) starting points look similar to the ones shown in Fig. 16a. However, CIFAR-10 [53] and ImageNet [54] yield much different results: those patterns look rather random compared to the results from LISA or MNIST. The reason might be that in CIFAR-10, images in the same category are still very different, such as two different cats, but in LISA, two images of STOP signs do not look as different as two cats.

For the experimental results of creation attacks, we see that as distances increase, success rates decrease a little (Fig. 15), but much better than the camera-aware attacks (Table I), because the optimization formulation helped find those optimal attack patterns with high confidence.

2) Alteration attacks: The emulated and experimental results of alteration attacks are shown in Fig. 15. Compared with creation attacks, alteration attacks perform a bit worse, especially for large distances (three meters or further). This is because the classifier also “sees” the benign image in the background and tends to classify the entire image as the benign class. Moreover, the alignment of attack patterns and the benign signs is imperfect. However, when we compare Fig. 15 with Table I for camera-aware alteration attacks, we can see large

Figure 16: System-aware attack pattern examples.

improvements. Fig. 16b provides an example of system-aware alteration attacks in the perception domain, which were trying to alter the (printed) STOP sign into other signs: they look “blue” as the channel model predicted. The fifth column is not showing as it is STOP.

A misclassification matrix of emulated alteration attacks at is given in Table II. The overall attack success rate was 75%. Each cell denotes the success rate of altering a benign class (actual) into a target class (predicted). Most of them are 100%, but the SCHOOL sign, for example, was the most difficult to perturb into (the 3rd column) and yet not that hard to perturb from (the 3rd row), probably because it is in green (RGB: 0-255-0) and in an opened-envelope shape, while all the others are either red (255-0-0) or yellow (255-255-0) colors, and either polygon or rectangle shapes.

B. Attack Robustness

We evaluate the robustness of our attacks in terms of different datasets, environments, and cameras. 1) Different image datasets: Here we evaluate our system-aware attacks on two other datasets, CIFAR-10 [53] and ImageNet [54], by emulation only because previous results show that our attack emulation yields similar success rates as experimental results. a) CIFAR-10: The network architecture and model hyper parameters are shown in Table VII and Table VIII in Ap-

Figure 17: System-aware attacks on CIFAR-10 and ImageNet

pendix A, which are identical to [29]. The network was trained with the distillation defense [55] so that we can evaluate the robustness of our attacks in terms of adversarial defenses. A classification accuracy of 80% was achieved. The evaluation procedure is similar to Algorithm 1. Results are shown in Fig. 17. The overall trend is similar to the LISA dataset, but the success rates are higher. The reason might still be the large variation within one class (Section VI-A1), so that the CIFAR-10 classifier is not as sure about one class as the LISA classifier is, hence is more vulnerable to our attacks. b) ImageNet: We used a pre-trained Inception V3 neural network [56] for the ImageNet dataset to evaluate the attack robustness against large networks. Since the pre-trained network can recognize 1000 classes, we did not iterate all of them (similar to [29]). Instead, for alteration attacks, we randomly picked ten benign images from the validation set, and twenty random target classes, while for creation attacks, the “benign” images were purely black. Results are given in Fig. 17. For high resolutions (), the attack success rates were nearly 100%. But as soon as the resolutions went down to or below, the rates decreased sharply. The reason might be that in order to mount successful targeted attacks on a 1000-class image classifier, a large number of degrees of freedom are required. or lower resolutions plus three color channels might not be enough to accomplish targeted attacks. To verify this, we also evaluated untargeted alteration attacks on ImageNet. Results show that when the resolutions are or , the success rates are 50% or 80%, respectively. But as soon as the resolutions go to or above, the success rates reach 100%. Lastly, similar to CIFAR-10, system-aware attacks

Table II: Emulated system-aware obfuscation attacks: misclas- sification matrix at -resolution. Predicted

Figure 18: Outdoor experiment setup

on ImageNet were more successful than on LISA, because of the high variation within one class. 2) Outdoor experiments: In order to evaluate system-aware attacks in a real-world environment, we also conducted experiments outdoor (Fig. 18), where the camera was put on the hood of a vehicle that was about to pass an intersection with a STOP sign. The attacker’s projector was placed on the right curb, and it was about four meters away from the camera. The experiments were done at noon, at dusk and at night (with the vehicle’s front lights on) to examine the effects of ambient light on attack efficacy. The illuminances were , , and 30 lx, respectively. The experiments at noon were unsuccessful due to the strong sunlight. Although more powerful projectors [57] could be acquired, we argue that a typical projector is effective in dimmer environments (e.g., cloudy days, at dawn, dusk, and night, or urban areas where buildings cause shades), which accounts for more than half of a day. See Sec. VII-A for more discussion on ambient lighting conditions. Results (Tab. III) of the other cases show that the success rates are 30% lower than our in-lab experiments (the fourmeter case from Fig. 15), because we used our in-lab channel model directly in the road experiments without retraining it, and also the environmental conditions are more unpredictable. Moreover, the attack rates on altering some classes (e.g., the STOP sign) into three other signs (e.g., YIELD) were 100%, which is critical as an attacker can easily prevent an autonomous vehicle from stopping at a STOP sign. 3) Different cameras: Previously, we conducted GhostImage attacks on Aptina MT9M034 camera [43] designed for autonomous driving. Here, we evaluate two other cameras, an Aptina MT9V034 [58] with a simpler lens design, and a Ring indoor security camera [47] for surveillance applications. a) Aptina MT9V034: We mounted system-aware creation attacks against the same camera-based object classification system as in Section VI-A but we replaced the camera with

Table III: Outdoor alteration attack success rates

the Aptina MT9V034 camera. Since this camera has a smaller aperture size and also a simpler lens design than Aptina MT9M034, for a distance of one meter, only -resolution attack patterns could be achieved (previously we had at one meter). We did not train a new channel model for this camera, so the attack success rate at one meter was only 75%, which is 25% lower than the Aptina MT9M034 camera. As the distances increased up to four meters, creation attacks yielded success rates as 46.25%, 33.75%, and 12.5%, respectively. Another reason why the overall success rate was lower is that even though the data sheet of Aptina MT9V034 [58] states that the camera also has the auto exposure control feature, we could not enable the feature in our experiments. In other words, system-aware creation attacks did not benefit from the exposure control. This, on the other hand, indicates the robustness of GhostImage attacks: Even without taking advantage of exposure control, the attacks were still effective.

b) Ring indoor security camera: We tested GhostImage untargeted attacks against a Ring indoor security camera [47] on the ImageNet dataset. To demonstrate that our attacks can be applied to surveillance scenarios, we assume the camera would issue an intrusion warning if a specific object type [59] is detected by the Inception V3 neural network [56]. The attacker’s goal is to change an object for an intruder class to a non-intruder class. However, we could not find “human”, “person” or “people”, etc. in the output classes, we instead used five human related items (such as sunglasses) as the benign classes. We found six images from the validation set of ImageNet, of which top-1 classification results are one of those five benign classes. The six images were displayed on a monitor. For each benign image, we calculated ten alternative attack patterns (the highest resolution at one meter by the Ring camera). Results show that our attacks achieve an overall untargeted attack success rate of 100% (Tab. IV).

In this section, we discuss practical challenges to GhostImage attacks, speculate as to effective countermeasures, and outline variations on the original attacks.

A. Practicality of GhostImage Attacks

Moving targets and alignment: The overlap of ghosts and objects of interest in images must be nearly complete for the attacks to succeed. In the cases of a moving camera (e.g., one mounted to a vehicle), the attacker needs to be able to accurately track the movement of the targeted camera, otherwise the

Table IV: GhostImage untargeted alteration attacks against Ring camera on ImageNet dataset in perception domain

attacker can only sporadically inject ghosts. Note that, although aiming (or tracking) moving targets is generally challenging in remote sensor attacks (e.g., the AdvLiDAR attack [16] assumes the attacker can achieve this via camera-based object detection and tracking), existing works [22], [35] have demonstrated the feasibility of tracking cameras and then neutralizing them. This paper’s main goal is to propose a new category of camera attacks, which enables an attacker to inject arbitrary patterns.

Conspicuousness: The light bursts around the light source in Figures 1 and 8 may raise stealthiness concerns about our attacks. However, according to our analysis in Sec. IV-B, such bursts can actually be eliminated because the light source can be outside of view (See Fig. 19 and [36]). Even the light source has to be in the frame (due to the lens configuration), we argue that a camera-based object classification system used in autonomous systems generally make decisions without human input (for example, in a Waymo self-driving taxi [2], no human driver is required, or in a Tesla car [3], real-time images would not typically be displayed). Additionally, the attack beam is so concentrated that only the victim camera can observe it while other human-beings (e.g., pedestrians) cannot (Fig. 18). Finally, the light source only needs to be on for a short amount of time, as a few tampered frames can cause incorrect actions [60].

Projectors, lenses, and attack distances: Based on our model (Eq. 7) and experiments (Tab. IX), the illuminance on the camera from the projector would better be 4/3 of the part from ambient illuminance (to achieve a success rate of 100%). Since Illuminance Luminance , in order to carry out an attack during sunny days (typically with Illuminance ), a typical projector (e.g., [61] with Luminance ) should work with a telephoto lens [62] (with a throwing radio 100) at a distance of one meter. For longer distances or brighter backgrounds, one can either acquire a more powerful projector (e.g., [57] with ), or combine multiple lenses to achieve much larger throwing ratios (e.g., two Optela lenses [62] yield 200, etc.), or both.

Ghost effect dependence: There are several challenges an attacker needs to overcome to launch GhostImage attacks. First, the attacks rely largely on ghost effects; if ghosts cannot be induced, or if they are not significant enough, the attacks might be infeasible against a given camera (lens). However, this is unlikely because these effects occur in most cameras (e.g., Apple iPhones [63], [64]). Moreover, no “flare-free” lens exists to the best of our knowledge (even with anti-glare coatings). In addition, if ghost effects are unavailable to the attacker there are other optics effects available, such as blooming effects [65], that can also be leveraged to produce GhostImage-like attacks.

Knowledge of the targeted system We assume that both types of attackers know about the camera matrix M and color calibration matrix . We note that the attacks can still be effective without such knowledge but with it the attacks can be more efficient. For example, the attacker may choose to lower their attack success expectation but the probability of successful attack may still be too high for potential victims to bear (e.g., a success rate of only 10% might be unacceptable for reasons of safety in automated vehicles). This challenge can be largely eliminated if the attacker is able to purchase a camera of the same, or similar, model as used in the targeted system and use it to derive the matrices. Although the duplicate camera may not be exactly the same to the target one, the channel model would still be in the same form with approximate, probably fine-tuned parameters (via retraining), thanks to the generality of our channel model. Lastly, assuming white-box knowledge on sensors is widely adopted and accepted in the literature, e.g., the AdvLiDAR attack [16]. Also, we assume white-box attacks on the neural network, though this assumption can be eliminated by leveraging the transferability of adversarial examples [66]–[69].

Object detection: We have assumed that the object detector can crop out the region of the image which contains the projected ghost pattern(s). Though it cannot be guaranteed that an object detector will automatically include the ghost patterns, we note that a GhostImage attacker could design ghost patterns that cause an object detector to include them [32], [70] and, at the same time, the cropped image would fool the subsequent object classifier.

B. Attack Variations

Should ghost effect be not available, we investigated alternative strategies that still allow an attacker to cause misclassi-ficaiton of objects of interest. inject adversarial noise, absent ghosts and/or flares, without placing the light source directly in front of the object, which would allow for straightforward detection of the attack. For example, using a beamsplitter that merges two light beams coming from two directions: one is the light reflected from the object the attacker wishes to obscure and the other is the light from the projector. The merged light beams enter the targeted camera as a superposition of the original object and the adversarial pattern, with the resulting image able to fool the classifier. Appendix D provides details on this attack vector and its efficacy.

Finally, while projectors provide an attacker with the greatest control over adversarial patterns, and hence the ability to spoof complex objects, we have found that RGB lasers [71] can be used at greater distances to spoof simple objects. It may also be possible for an attacker to use multi-laser systems, or even flashlights [72], to create complex patterns akin to the decorative lights displayed on Christmas trees [73].

C. Countermeasures

The most straightforward countermeasure to GhostImage attacks is flare elimination, either by using a lens hood [26], [74] or through flare detection. Lens hoods are generally not favored as they reduce the angle of view of the camera, which is unacceptable for many autonomous vehicle and surveillance applications. Note that there are so-called liquid lenses that can change its focal length by reshaping itself, which results in only one reflection path, hence fewer flares. However, such lenses have not been widely adopted yet [75].

Currently, we are working on a defense where we first identify all flares/ghosts in an image, and then detect if a ghost contains malicious patterns. However, the challenges include: First, ghosts are typically transparent thus hard to detect [76]; Second, natural ghosts are so common and varied that false positives can occur inevitably; Third, just as adversarial noise can be crafted to deceive neural networks an analogous procedure could be used to craft flares/ghosts that deceive flare/ghost detectors.

A complementary line of defense would be to make neural networks themselves robust to GhostImage attacks. Existing approaches against adversarial examples (e.g., [55], [77]–[79], etc.) are ill-suited for this task, however, as GhostImage attacks do not necessarily follow the constraints placed on traditional adversarial examples in that perturbations do not have to be bounded within a small norm, meanwhile these defenses were not designed for arbitrarily large perturbations.

Another complementary approach of defense is to exploit prior knowledge, such as GPS locations of signs, to make decisions, instead of only depending on real-time sensor perception (though this approach would not work for spontaneous appearance of objects, e.g., in the context of collision avoidance). Sensor redundancy/fusion could also be helpful: autonomous vehicles could be equipped with multiple cameras and/or other types of sensors, such as LiDARs and radars, which would at least increase the cost of the attack by requiring the attacker to target multiple sensors.

Since our attack spans two domains, in this section we review both sensor attacks and adversarial examples.

A. Sensor attacks

Perception in autonomous and surveillance systems occurs through sensors, which convert analog signals into digital ones that are further analyzed by computing systems. Recent work has demonstrated that the sensing mechanism itself is vulnerable to attack and that such attacks may be used to bypass digital protections [19], [20]. For example, anti-lock braking system (ABS) sensors have been manipulated via magnetic fields by Shoukry et al. [80], microphones have been subject to inaudible voice and light-based attacks [13], [81], and light sensors can be influenced via electromagnetic interference to report lighter or darker conditions [12]. The reader is referred to [19], [20] for a review of analog sensor attacks.

Existing remote attacks against cameras [15], [22], [23] are denial-of-service attacks and do not seek to compromise the object classifier as our GhostImage attacks do. Those attacks that do target object classification [30], [32], [82] are either digital or physical domain attacks (i.e., they need to modify the object of interest in this case a traffic sign or road pavement, physically or after the object has been captured by a camera) rather than perception domain attacks [19], [20]. Li et al. [34]’s attacks on cameras require attackers to place stickers on lenses, to which is generally hard to get access. Similarly, several light-based attacks [60], [83], [84] fall within the domain of physical attacks, as opposed to our perception domain attack, because these approaches illuminate the object of interest with visible or infrared light. We did not consider infrared noise in our attacks as it can be easily eliminated from visible light systems using infrared filters. Attacks on LiDAR systems [14]– [16], [85] are also related to this work; however, these attacks are considerably easier to carry out than our visible light-based attacks against cameras because attackers can directly inject adversarial laser pulses into LiDARs without worrying about blocking the object of interest.

B. Adversarial examples

State-of-the-art adversarial examples can be categorized as digital [28], [29], [33], [52], [66]–[68], [86]–[91], or physical domain attacks [30]–[32], [69], [83], [87], [92]–[96] in which objects of interest are physically modified to cause misclassification. The latter differs from GhostImage attacks in that we target the sensor (camera) without needing to physically modify any real-world object. Another line of work focuses on unrestricted adversarial examples [97]–[99], though they are limited in the digital domain.

In terms of defending neural networks from adversarial examples, be they physical or digital, schemes include modifying the network to be more robust [33], [55], [77]–[79], [100]– [105], while other defenses have focused on either detecting adversarial inputs [106]–[111] or transforming them into benign images [109], [112], [113], most of which are under the general assumption of bounded perturbations, hence are inapplicable to our attacks; while others could also be bypassed by being taken as constraints in the optimization formulation. As this work mainly focuses on sensor attacks, similar to [16], [30]–[32] we leave the validation of defenses as future work.

In this work we presented GhostImage attacks against camera-based object classifiers. Using common optical effects, viz. lens flare/ghost effects, an attacker is able to inject arbitrary adversarial patterns into camera images using a projector. To increase the efficacy of the attack, we proposed a projector-camera channel model that predicts the location of ghosts, the resolution of the patterns in ghosts, given the projector-camera arrangement, and accounts for exposure control and color calibration. GhostImage attacks also leverage adversarial examples generation techniques to find optimal attack patterns. We evaluated GhostImage attacks using three image datasets and in both indoor and outdoor environments on three cameras. Experimental results show that GhostImage attacks were able to achieve attack success rates as high as 100%, and also have potential impact on autonomous systems, such as self-driving cars and surveillance systems.

Here we present the architectures of two neural networks (Tables V and VII) and their hyper-parameters (Table VIII). The balanced LISA dataset is also detailed in Table VI.

Figure 19: There are ghosts but the light source is out of view.

Table V: Neural network architecture for LISA dataset

APPENDIX B DETAILED PROJECTOR-CAMERA MODEL PARAMETERS

Table IX lists all parameters of the projector-camera channel model. The color calibration matrix is

and the camera matrix is

APPENDIX C DIRECT PROJECTION USING AN ADDITIONAL LENS

Lemma C.1. However the additional lenses are placed, the image of noise cannot overlay with the image of the object without obscuring the object.

Proof. We are going to prove that even with an additional concave lens. See Figure 20 for a diagram, where a concave lens is placed between an object AB and a camera’s convex lens ’s focal length is and ’s is . The distance between and is . The noise source N is right upon A. From the perspective of is completely obfuscated by ; in other words, all light rays of AB that go through must at first go through . Both the object and the noise source share the same object distance to image of is the image of N. The distance between and is . The image of formed by is and . The distance between and is .

In order to solve a problem consisting of multiple lenses, we usually analyze each lens individually and sequentially. That is, we calculate the image formed by the first lens, then use that image as the input to the second lens.

Table VI: Balanced LISA dataset

Table VII: Neural network architecture for CIFAR-10 dataset

For , the input is AB, thus we have

and the magnification is calculated as

from which we know that does not overlap with . For , the input is cannot “see” AB directly because AB is completely obfuscated by ), thus we get

and the magnification is calculated as

from which we know that does not overlap with a result, no matter how we place the additional concave lens, we cannot apply the noise to the image of the object without obscuring the object. The proof for a convex lens follows the same logic.

See Figure 21a for a diagram of this method, where a beamsplitter is used to merge two light beams coming from two directions. The light beams coming from the object (marked in red) get reflected and transmitted, i.e., the beamsplitter does not obscure the object. The transmitted portions go into the camera, forming an image of the object. The light beams from

Table VIII: Training hyper-parameters

Table IX: Channel model parameter examples

the light noise source (marked in blue) also get reflected and transmitted. The reflected portions (instead of the transmitted portions) are captured by the camera, forming an image of the noise. Two images overlap as a potential adversarial example, depicted as a small magenta (red plus blue) stop sign in the camera.

An in-lab experimental setup is shown in Figure 21b. In this setup, we used the NEC projector [114] as the light noise source. Instead of using an expensive beamsplitter, we found out that a single piece of glass could also function like a beamsplitter. We placed a piece of white paper in front of the projector’s lens to reduce the projected image size (otherwise the projected image becomes too large even within a small throwing distance). This does not change the attack plausibility because the imager can clearly capture the noise pattern on the paper. These elements were placed in a way that the noise image (from the paper) would overlap with the image of the object from the view of the camera. A misclassification matrix is shown in Figure 22 where an overall attack success rate of 55% was achieved.

Figure 20: Noise N cannot overlap with the image of the object AB even with an additional concave lens.

Figure 21: The beamsplitter method setup.

[1] Uber. Self-driving car technology by uber. https://www.uber.com/us/en/ atg/technology/, 2020.

[2] Waymo. Waymo. https://waymo.com, 2020.

[3] Tesla. Autopilot. https://www.tesla.com/autopilot, 2020.

[4] Skydio. Skydio 2. skydio.com, 2020.

[5] Amazon. Prime air delivery. https://www.amazon.com/Amazon-Prime- Air/b?ie=UTF8&node=8037720011, 2020.

Figure 22: Misclassification matrix of the beamsplitting method at resolutions

[6] Google. Nest and google home. now under one roof. nest.com, 2020.

[7] Amazon. Ring. ring.com, 2020.

[8] Stephen Checkoway, Damon McCoy, Brian Kantor, Danny Anderson, Hovav Shacham, Stefan Savage, Karl Koscher, Alexei Czeskis, Franziska Roesner, Tadayoshi Kohno, et al. Comprehensive experimental analyses of automotive attack surfaces. In USENIX Security Symposium, volume 4, pages 447–462. San Francisco, 2011.

[9] Charlie Miller and Chris Valasek. A survey of remote automotive attack surfaces. black hat USA, 2014:94, 2014.

[10] Jonathan Petit and Steven E Shladover. Potential cyberattacks on automated vehicles. IEEE Transactions on Intelligent Transportation Systems, 16(2):546–556, 2014.

[11] Andrei Costin, Jonas Zaddach, Aurélien Francillon, and Davide Balzarotti. A large-scale analysis of the security of embedded firmwares. In 23rd {USENIX} Security Symposium ({USENIX} Security 14), pages 95–110, 2014.

[12] Jayaprakash Selvaraj, Gökçen Y Dayanıklı, Neelam Prabhu Gaunkar, David Ware, Ryan M Gerdes, Mani Mina, et al. Electromagnetic induction attacks against embedded systems. In Proceedings of the 2018 on Asia Conference on Computer and Communications Security, pages 499–510. ACM, 2018.

[13] Takeshi Sugawara, Benjamin Cyr, Sara Rampazzi, Daniel Genkin, and Kevin Fu. Light commands: Laser-based audio injection attacks on voice-controllable systems.

[14] Hocheol Shin, Dohyun Kim, Yujin Kwon, and Yongdae Kim. Illusion and dazzle: Adversarial optical channel exploits against lidars for automotive applications. In International Conference on Cryptographic Hardware and Embedded Systems, pages 445–467. Springer, 2017.

[15] Jonathan Petit, Bas Stottelaar, Michael Feiri, and Frank Kargl. Remote attacks on automated vehicles sensors: Experiments on camera and lidar. Black Hat Europe, 11:2015, 2015.

[16] Yulong Cao, Chaowei Xiao, Benjamin Cyr, Yimeng Zhou, Won Park, Sara Rampazzi, Qi Alfred Chen, Kevin Fu, and Z Morley Mao. Adversarial sensor attack on lidar-based perception in autonomous driving. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 2267–2281. ACM, 2019.

[17] Yunmok Son, Hocheol Shin, Dongkwan Kim, Youngseok Park, Juhwan Noh, Kibum Choi, Jungwoo Choi, and Yongdae Kim. Rocking drones with intentional sound noise on gyroscopic sensors. In 24th {USENIX} Security Symposium ({USENIX} Security 15), pages 881–896, 2015.

[18] Qiben Yan, Kehai Liu, Qin Zhou, Hanqing Guo, and Ning Zhang. Surfingattack: Interactive hidden attack on voice assistants using ultrasonic guided wave. In Network and Distributed Systems Security (NDSS) Symposium, 2020.

[19] C. Yan, H. Shin, C. Bolton, W. Xu, Y. Kim, and K. Fu. Sok: A minimalist approach to formalizing analog sensor security. In 2020 IEEE Symposium on Security and Privacy (SP), pages 480–495, Los Alamitos, CA, USA, may 2020. IEEE Computer Society.

[20] Ilias Giechaskiel and Kasper Bonne Rasmussen. Taxonomy and challenges of out-of-band signal injection attacks and defenses. IEEE Communication Surveys & Tutorials, 2020.

[21] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.

[22] Khai N Truong, Shwetak N Patel, Jay W Summet, and Gregory D Abowd. Preventing camera recording by designing a capture-resistant environment. In International conference on ubiquitous computing, pages 73–86. Springer, 2005.

[23] Chen Yan, Wenyuan Xu, and Jianhao Liu. Can you trust autonomous vehicles: Contactless attacks against sensors of self-driving vehicle. DEF CON, 24, 2016.

[24] Evan Ribnick, Stefan Atev, Osama Masoud, Nikolaos Papanikolopoulos, and Richard Voyles. Real-time detection of camera tampering. In 2006 IEEE International Conference on Video and Signal Based Surveillance, pages 10–10. IEEE, 2006.

[25] Qingquan Li, Long Chen, Ming Li, Shih-Lung Shaw, and Andreas Nüchter. A sensor-fusion drivable-region and lane-detection system for autonomous vehicle navigation in challenging road scenarios. IEEE Transactions on Vehicular Technology, 63(2):540–555, 2013.

[26] Yaopey. How to deal with lens flare. www.fotographee.com/how-to- deal-with-lens-flare, 2019.

[27] Matthias B. Hullin, Elmar Eisemann, Hans-Peter Seidel, and Sungkil Lee. Physically-based real-time lens flare rendering. ACM Trans. Graph. (Proc. SIGGRAPH 2011), 30(4):108:1–108:9, 2011.

[28] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.

[29] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Proceedings of the IEEE Symposium on Security and Privacy, pages 39–57. IEEE, 2017.

[30] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1625–1634, 2018.

[31] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 1528–1540. ACM, 2016.

[32] Yue Zhao, Hong Zhu, Ruigang Liang, Qintao Shen, Shengzhi Zhang, and Kai Chen. Seeing isn’t believing: Towards more robust adversarial attack against real world object detectors. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1989–2004. ACM, 2019.

[33] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.

[34] Juncheng B Li, Frank R Schmidt, and J Zico Kolter. Adversarial camera stickers: A physical camera attack on deep learning classifier. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, 2019.

[35] KYLE MIZOKAMI. China could blind u.s. satellites with lasers. https://www.popularmechanics.com/military/weapons/a29307535/ china-satellite-laser-blinding/, 2019.

[36] Gunawan Kartapranata. Lens flare at borobudur stairs kala arches, 2010.

[37] Sungkil Lee and Elmar Eisemann. Practical real-time lens-flare rendering. In Computer Graphics Forum, volume 32, pages 1–6. Wiley Online Library, 2013.

[38] Benjamin Steinert, Holger Dammertz, Johannes Hanika, and Hendrik PA Lensch. General spectral camera lens simulation. In Computer Graphics Forum, volume 30, pages 1643–1654. Wiley Online Library, 2011.

[39] Patricia Vitoria and Coloma Ballester. Automatic flare spot artifact detection and removal in photographs. Journal of Mathematical Imaging and Vision, 61(4):515–533, 2019.

[40] Hsien-Che Lee. Introduction to color imaging science. Cambridge University Press, 2005.

[41] Spencer Cox. What is exposure? (a beginnerâ ˘A´Zs guide). https:// photographylife.com/what-is-exposure, 2019.

[42] Richard Szeliski. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.

[43] ON semiconductor. MT9M034 1/3-Inch CMOS Digital Image Sensor, 2017.

[44] NEC. NP Installation Series User’s Manual, 10 2007.

[45] Canon. Telephoto zoom ef-s 55-250mm. https://www.usa.canon.com/ internet/portal/us/home/products/details/lenses/ef/telephoto-zoom/ef-s- 55-250mm-f-4-5-6-is-ii, 2019.

[46] NEC. Np05zl, 4.62â ˘A¸S7.02:1 zoom lens. https://www.necdisplay.com/ p/optional-lenses/np05zl, 2019.

[47] Ring. Indoor security cameras. https://shop.ring.com/collections/ security-cams#indoor, 2019.

[48] Andreas Mogelmose, Mohan Manubhai Trivedi, and Thomas B Moeslund. Vision-based traffic sign detection and analysis for intelligent driver assistance systems: Perspectives and survey. IEEE Transactions on Intelligent Transportation Systems, 13(4):1484–1497, 2012.

[49] Dr. Meter. Digital illuminance meter. https://www.amazon.com/gp/ product/B07LF4BT8V, 2019.

[50] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.

[51] Nvidia. NVIDIA TESLA P100 GPU ACCELERATOR, 2016.

[52] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pages 372–387. IEEE, 2016.

[53] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

[54] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.

[55] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of the IEEE Symposium on Security and Privacy, pages 582–597. IEEE, 2016.

[56] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.

[57] Barco. Xdl-4k75. https://www.barco.com/en/product/xdl-4k75, 2019.

[58] ON semiconductor. MT9V034 1/3-Inch Wide-VGA CMOS Digital Image Sensor, 2017.

[59] Ring. Standard and advanced motion detection systems used in ring devices. https://support.ring.com/hc/en-us/articles/115005914666- Standard-and-Advanced-Motion-Detection-Systems-Used-in-Ring- Devices, 2020.

[60] Ben Nassi, Dudi Nassi, Raz Ben-Netanel, Yisroel Mirsky, Oleg Drokin, and Yuval Elovici. Phantom of the adas: Phantom attacks on driverassistance systems.

[61] Epson. Pro l1490u wuxga 3lcd laser projector. https://epson.com/For- Work/Projectors/Large-Venue/Pro-L1490U-WUXGA-3LCD-Laser- Projector-with-4K-Enhancement-and-Lens/p/V11HA16020, 2019.

[62] Opteka. Opteka 650-1300mm telephoto zoom lens. https://www.amazon.com/Opteka-650-1300mm-1300-2600mm- Telephoto-Digital/dp/B001VDLZIG, 2019.

[63] Apple. iphone 8 plus horrible lens flare and reflections. https: //discussions.apple.com/thread/8118139, 2017.

[64] Apple. iphone 11 proâ ˘A´Zs irritating lens flare problem. https://www.reddit.com/r/apple/comments/da6qft/iphone_11_pros_ irritating_lens_flare_problem/, 2019.

[65] Wikipedia. Bloom (shader effect). https://en.wikipedia.org/wiki/Bloom_ (shader_effect), 2020.

[66] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016.

[67] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pages 506–519. ACM, 2017.

[68] Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Practical black-box attacks on deep neural networks using efficient query mechanisms. In European Conference on Computer Vision, pages 158–174. Springer, 2018.

[69] Yuxuan Chen, Xuejing Yuan, Jiangshan Zhang, Yue Zhao, Shengzhi Zhang, Kai Chen, and XiaoFeng Wang. Devilâ ˘A´Zs whisper: A general approach for physical adversarial attacks against commercial black-box speech recognition devices. In 29th USENIX Security Symposium (USENIX Security 20), 2020.

[70] Dawn Song, Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramer, Atul Prakash, and Tadayoshi Kohno. Physical adversarial examples for object detectors. In 12th {USENIX} Workshop on Offensive Technologies ({WOOT} 18), 2018.

[71] Laser World. Rgb laser. https://www.laserworld.com/en/show-laser- light-faq/glossary-definitions/88-r/2549-rgb-laser.html.

[72] Christian Carlberg. Hexbright, an open source light. https://www.kickstarter.com/projects/christian-carlberg/hexbright- an-open-source-light.

[73] https://linx.li/wl9efae2.jpg.

[74] Anna Altez. Lens flare: How to reduce or avoid it? https://www. photopoly.net/lens-flare-how-to-reduce-or-avoid-it/, 2011.

[75] Edmond Optics. Introduction to liquid lenses. https: //www.edmundoptics.com/knowledge-center/application- notes/imaging/introduction-to-liquid-lenses/.

[76] Yichao Xu, Hajime Nagahara, Atsushi Shimada, and Rin-ichiro Taniguchi. Transcut: Transparent object segmentation from a light-field image. In Proceedings of the IEEE International Conference on Computer Vision, pages 3442–3450, 2015.

[77] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant

to adversarial attacks. In International Conference on Learning Representations, 2018.

[78] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In IEEE Symposium on Security and Privacy (S&P), 2019.

[79] Tong Wu, Liang Tong, and Yevgeniy Vorobeychik. Defending against physically realizable attacks on image classification. In 8th International Conference on Learning Representations (ICLR), 2020.

[80] Yasser Shoukry, Paul Martin, Paulo Tabuada, and Mani Srivastava. Noninvasive spoofing attacks for anti-lock braking systems. In International Workshop on Cryptographic Hardware and Embedded Systems, pages 55–72. Springer, 2013.

[81] Guoming Zhang, Chen Yan, Xiaoyu Ji, Tianchen Zhang, Taimin Zhang, and Wenyuan Xu. Dolphinattack: Inaudible voice commands. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS â ˘A´Z17, page 103â ˘A¸S117, New York, NY, USA, 2017. Association for Computing Machinery.

[82] Alesia Chernikova, Alina Oprea, Cristina Nita-Rotaru, and BaekGyu Kim. Are self-driving cars secure? evasion attacks against deep neural networks for steering angle prediction. In IEEE Security and Privacy Workshop on IoT. IEEE, 2019.

[83] Zhe Zhou, Di Tang, Xiaofeng Wang, Weili Han, Xiangyu Liu, and Kehuan Zhang. Invisible mask: Practical attacks on face recognition with infrared. arXiv preprint arXiv:1803.04683, 2018.

[84] Luan Nguyen, Sunpreet S. Arora, Yuhang Wu, and Hao Yang. Adversarial light projection attacks on face recognition systems: A feasibility study, 2020.

[85] James Tu, Mengye Ren, Siva Manivasagam, Ming Liang, Bin Yang, Richard Du, Frank Cheng, and Raquel Urtasun. Physically realizable adversarial examples for lidar object detection, 2020.

[86] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2574–2582, 2016.

[87] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security, pages 99–112. Chapman and Hall/CRC, 2018.

[88] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765–1773, 2017.

[89] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In Proceedings of the IEEE conference on learning representations, 2018.

[90] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In Proceedings of the International Conference on Learning Representations, 2016.

[91] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decisionbased adversarial attacks: Reliable attacks against black-box machine learning models. In Proceedings of the IEEE conference on learning representations, 2018.

[92] Chawin Sitawarin, Arjun Nitin Bhagoji, Arsalan Mosenia, Prateek Mittal, and Mung Chiang. Rogue signs: Deceiving traffic sign recognition with malicious ads and logos. In IEEE Security and Privacy Workshop on Deep Learning and Security. IEEE, 2018.

[93] Zuxuan Wu, Ser-Nam Lim, Larry Davis, and Tom Goldstein. Making an invisibility cloak: Real world adversarial attacks on object detectors. arXiv preprint arXiv:1910.14667, 2019.

[94] Stepan Komkov and Aleksandr Petiushko. Advhat: Real-world adversarial attack on arcface face id system. arXiv preprint arXiv:1908.08705, 2019.

[95] Ranjie Duan, Xingjun Ma, Yisen Wang, James Bailey, A. K. Qin, and Yun Yang. Adversarial camouflage: Hiding physical-world attacks with natural styles, 2020.

[96] Takami Sato, Junjie Shen, Ningfei Wang, Yunhan Jack Jia, Xue Lin, and Qi Alfred Chen. Security of deep learning based lane keeping system under physical-world adversarial attack, 2020.

[97] Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Advances in Neural Information Processing Systems, pages 8312–8323, 2018.

[98] Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1614–1619, 2018.

[99] Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and D. A. Forsyth. Unrestricted adversarial examples via semantic manipulation. In International Conference on Learning Representations, 2020.

[100] Guneet S. Dhillon, Kamyar Azizzadenesheli, Jeremy D. Bernstein, Jean Kossaifi, Aran Khanna, Zachary C. Lipton, and Animashree Anandkumar. Stochastic activation pruning for robust adversarial defense. In International Conference on Learning Representations, 2018.

[101] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations, 2018.

[102] Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pages 369–385, 2018.

[103] Eric Wong and J Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, 2018.

[104] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. In International Conference on Learning Representations, 2018.

[105] Xiaoyu Cao and Neil Zhenqiang Gong. Mitigating evasion attacks to deep neural networks via region-based classification. In Proceedings of the 33rd Annual Computer Security Applications Conference, pages 278–287. ACM, 2017.

[106] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In Network and Distributed Systems Security Symposium (NDSS), 2018.

[107] Dan Hendrycks and Kevin Gimpel. Early methods for detecting adversarial images. 2017.

[108] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In Proceedings of the IEEE conference on learning representations, 2017.

[109] Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 135–147. ACM, 2017.

[110] Shiqing Ma, Yingqi Liu, Guanhong Tao, Wen-Chuan Lee, and Xiangyu Zhang. Nic: Detecting adversarial samples with neural network invariant checking. In Network and Distributed System Security Symposium, 2019.

[111] Feiyang Cai, Jiani Li, and Xenofon Koutsoukos. Detecting adversarial examples in learning-enabled cyber-physical systems using variational autoencoder for regression. In Workshop on Assured Autonomous Systems, 2020.

[112] Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations, 2018.

[113] Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations, 2018.

[114] NEC. NP Installation Series Specification Sheet, 11 2009.