The goal of many empirical analyses is to understand the causal effect of a treatment, such as participation in a training program or a government policy, on economic and other outcomes. Such analyses are often complicated by the fact that treatments or policies are rarely randomly assigned. The lack of true random assignment has led to the adoption of a variety of quasi-experimental approaches to estimating treatment effects that are based on observational data. Such approaches include instrumental variable (IV) methods in cases where treatment is not randomly assigned but there is some other external variable, such as eligibility for receipt of a government program or service, that is either randomly assigned or the researcher is willing to take as exogenous conditional on the right set of control variables (or simply controls). Another common approach is to assume that the treatment variable itself may be taken as exogenous after conditioning on the right set of controls which leads to regression or matching based methods, among others, for estimating treatment effects.

A practical problem empirical researchers face when trying to estimate treatment effects is deciding what conditioning variables to include. When the treatment variable or instrument is not randomly assigned, a researcher must choose what needs to be conditioned on to make the argument that the instrument or treatment is exogenous plausible. Typically, economic intuition will suggest a set of variables that might be important to control for but will not identify exactly which variables are important or the functional form with which variables should enter the model. While less crucial to identifying treatment effects, the problem of selecting controls also arises in situations where the key treatment or instrumental variables are randomly assigned. In these cases, a researcher interested in obtaining precisely estimated policy effects will also typically consider including additional controls to help absorb residual variation. As in the case where including controls is motivated by a desire to make identification of the treatment effect more plausible, one rarely knows exactly which variables will be most useful for accounting for residual variation. In either case, the lack of clear guidance about what variables to use presents the problem of selecting controls from a potentially large set including raw variables available in the data as well as interactions and other transformations of these variables.

In this paper, we consider estimation of the effect of an endogenous binary treatment, D, on an outcome, Y , in the presence of a binary instrumental variable, Z, in settings with very many potential controls, f(X). Allowing many potential controls expressly covers both the case where there are simply many controls (where f(X) = X)) and the case where there are many technical controls f(X) generated as transformations such as powers, b-splines, or interactions of raw controls,along with combinations of the two cases. The notation f(X) naturally accommodates these cases, and we call f(X) the controls regardless of the case. We allow for fully heterogeneous treatment effects and thus focus on estimation of causal quantities that are appropriate in heterogeneous effects settings such as the local average treatment effect (LATE) or the local quantile treatment effect (LQTE). We focus our discussion on the endogenous case where identification is obtained through the use of an instrumental variable, but all results carry through to the exogenous case where the treatment is taken as exogenous unconditionally or after conditioning on sufficient controls by simply replacing the instrument with the treatment variable in the estimation and inference methods and in the formal results. In the latter case, LATE reduces to the average treatment effect (ATE) and LQTE to the quantile treatment effect (QTE).

The methodology for estimating treatment effects we consider allows for cases where the number of potential controls, p := dim f(X), is much larger than the sample size, n. Of course, informative inference about causal parameters cannot proceed allowing for without further restrictions. We impose sufficient structure through the assumption that reduced form relationships such as the conditional expectations E] are approximately sparse. Intuitively, approximate sparsity imposes that these reduced form relationships can be represented up to a small approximation error as a linear combination, possibly inside of a known link function such as the logistic function, of a number of the variables in f(X) whose identities are a priori unknown to the researcher. This assumption allows us to use methods for estimating models in high-dimensional sparse settings that are known to have good prediction properties to estimate the fundamental reduced form relationships. We may then use these estimated reduced form quantities as inputs to estimating the causal parameters of interest. Approaching the problem of estimating treatment effects within this framework allows us to accommodate the realistic scenario in which a researcher is unsure about exactly which confounding variables or transformations of these confounds are important and so must search among a broad set of controls.

Valid inference following model selection is non-trivial. Direct application of usual inference procedures following model selection does not provide valid inference about causal parameters even in low-dimensional settings, such as when there is only a single control, unless one assumes sufficient structure on the model that perfect model selection is possible. Such structure can be restrictive and seems unlikely to be satisfied in many economic applications. For example, a typical condition that allows perfect model selection is the “beta-min” condition, which requires that all but a small number of coefficients are exactly zero and that the non-zero coefficients are all large enough that they can be distinguished from zero with probability very near one in finite samples. Such a condition rules out the possibility that there may be some variables which have moderate, but non-zero, partial effects. Ignoring such variables may result in large omitted variables bias that has a substantive impact on estimation and inference regarding individual model parameters; see Leeb and P¨otscher (2008a; 2008b); P¨otscher (2009); and Belloni, Chernozhukov, and Hansen (2013a; 2014a).

The first main contribution of this paper is providing inferential procedures for key parameters used in program evaluation that are theoretically valid within approximately sparse models allowing for imperfect model selection. Our procedures build upon Belloni et al. (2010) and Belloni et al. (2012), who were the first to demonstrate in a highly specialized context, that valid inference can proceed following model selection allowing for model selection mistakes under two conditions. We formulate and extend these two conditions to a rather general moment-condition framework (e.g., Hansen (1982) and Hansen and Singleton (1982)) as follows. First, estimation should be based upon “orthogonal” moment conditions that are first-order insensitive to changes in the values of nuisance parameters that will be estimated using high-dimensional methods. Specifically, if the target parameter value is identified via the moment condition

where is a function-valued nuisance parameter estimated via a model-selection or regularization method, one needs to use a moment function, , such that the corresponding moment condition is orthogonal with respect to perturbations of . More formally, the moment condition should satisfy the Neyman orthogonality condition

where is a functional derivative operator with respect to h restricted to directions of possible deviations of estimators of Second, one needs to ensure that the model selection mistakes occurring in the estimation of nuisance parameters are uniformly “moderately” small with respect to the underlying model. Specifically, we will require that the nuisance parameter is estimated at the rate ), which ensures small bias, and that the estimator takes values in a space whose entropy does not grow too fast, which ensures no overfitting. In this paper, we establish that building estimators based upon moment conditions with the orthogonality condition (1.2) holding ensures that crude estimation of via post-selection or other regularization methods has an asymptotically negligible effect on the estimation of in general frameworks. It then follows that we can form a regular, root-n consistent estimator of , uniformly with respect to the underlying model.

In the endogenous treatment effects setting, we build moment conditions satisfying (1.2) from the efficient influence functions for certain reduced form parameters, building upon Hahn (1998). We illustrate how orthogonal moment conditions coupled with methods developed for forecasting in high-dimensional approximately sparse models can be used to estimate and obtain valid inferential statements about a wide variety of structural/treatment effects. We formally demonstrate the uniform validity of the resulting inference within a broad class of approximately sparse models including models where perfect model selection is theoretically impossible. An important feature of our main theoretical results is that they cover the use of variable selection for functional response -penalized methods. Functional response data arises, for example, when one is interested in the LQTE at not just a single quantile but over a range of quantile indices. Considering this case then necessitates looking at the functional dependent variable u denotes various levels that Y can cross. Treating such functional response data allows us to provide a unified inference procedure for interesting quantities such as the (local) distributional and quantile effects of the treatment, including simpler important parameters such as LQTE at a given quantile as a special case.

The second main contribution of this paper is providing a general set of results for uniformly valid estimation and inference methods in moment-condition problems, arising in structural analysis in econometrics and other data sciences. These results are useful not only for establishing the properties of treatment effects estimators developed here, but they are also useful for attacking a wide range of problems in structural econometrics. For example, Chernozhukov et al. (2015a) provide estimates of parameters characterizing a simple structural demand model based loosely on the analysis in Berry et al. (1995) using the framework developed here; see also Chernozhukov et al. (2015b). A key element to our establishing uniform validity of post-regularization inference is again the use of Neyman orthogonal moment conditions. In the general framework we consider, we may have (a continuum of) target parameters identified via (a continuum of) moment conditions that involve (a continuum of) nuisance functions that will be estimated via Lasso, Post-Lasso, or some other high-quality machine learning method. Our general theory expressly allows for a wide variety of traditional and machine learning methods, including those that do not rely on approximate sparsity, as long as the methods

By “not overfitting” we mean that the entropy of the function classes containing the realizations of the estimator of the nuisance function/parameter does not increase too rapidly with the sample size. This second condition can only be verified analytically, but can be avoided by the use of various data splitting methods. For example, we can set aside a vanishing fraction of the data to estimate the nuisance parameter, as in Bickel (1982), or employ cross-fitting, as in Belloni et al. (2010, 2012) and Chernozhukov et al. (2016). Either scheme ensures that there is no asymptotic efficiency loss from data-splitting. We refer the reader to Chernozhukov et al. (2016) for a detailed discussion and analysis of cross-fitting in connection to inference on ATE and other causal parameters using machine learning methods for high-dimensional data.

These results contain the results on treatment effects relevant for program evaluation, particularly the results for distributional and quantile effects, as a leading special case. These results are also immediately useful in other contexts such as nonseparable quantile models as in Chernozhukov and Hansen (2005), Chernozhukov and Hansen (2006), Chesher (2003), and Imbens and Newey (2009); semiparametric and partially identified models as in Escanciano and Zhu (2013); and many others. In our results, we first establish a functional central limit theorem for the continuum of target parameters and show that this functional central limit theorem holds uniformly in a wide range of data-generating processes P with approximately sparse continua of nuisance functions. Second, we establish a functional central limit theorem for the multiplier bootstrap that resamples the first order approximations to the standardized estimators and demonstrate its uniform-in-P validity. These uniformity results build upon and complement those given in Romano and Shaikh (2012) for the empirical bootstrap. Third, we establish a functional delta method for smooth functionals of the continuum of target parameters and a functional delta method for the multiplier bootstrap of these smooth functionals, both of which hold uniformly in P, using an appropriately strengthened notion of Hadamard differentiability. All of these results are new and are of independent interest outside of the treatment effects focus of this paper.

We illustrate the use of our methods by estimating the effect of 401(k) eligibility and 401(k) participation on measures of accumulated assets as in Chernozhukov and Hansen (2004).to Chernozhukov and Hansen (2004), we provide estimates of ATE and QTE of 401(k) eligibility and of LATE and LQTE of 401(k) participation. We differ from this previous work by using the high-dimensional methods developed in this paper to allow ourselves to consider a broader set of controls than has previously been considered. We find that 401(k) participation has a moderate impact on accumulated financial assets at low quantiles while appearing to have a much larger impact at high quantiles. Interpreting the quantile index as “preference for savings” as in Chernozhukov and Hansen (2004), this pattern suggests that 401(k) participation has little causal impact on the accumulated financial assets of those with low desire to save but a much larger impact on those with stronger preferences for saving. It is interesting that these results are similar to those in Chernozhukov and Hansen (2004) despite allowing for a much richer set of controls.

Links to the literature. The Neyman orthogonality condition embodied in (1.2) has a long history in statistics and econometrics. For example, this type of orthogonality was used by Neyman (1979) in low-dimensional settings to deal with crudely estimated parametric nuisance parameters. See also Newey (1990), Andrews (1994b), Newey (1994), Robins and Rotnitzky (1995), and Linton (1996) for the use of this condition in semi-parametric problems.

To the best of our knowledge, Belloni et al. (2010) and Belloni et al. (2012) were the first to use the orthogonality (1.2) to expressly address the question of the uniform post-selection inference without imposing “beta-min” conditions, either in high-dimensional settings with or in low-dimensional settings with . They applied it to the specific problem of the linear instrumental variables model with many instruments where the nuisance function is the optimal instrument estimated by Lasso or Post-Lasso methods and is the coefficient of the endogenous regressor. Belloni et al. (2013a) and Belloni et al. (2014a) also exploited this approach to develop a doubleselection method that yields valid post-selection inference on the parameters of the linear part of a partially linear model and on average treatment effects when the treatment is binary and exogenous conditional on controls in both the Subsequently, Farrell (2015) extended the results of Belloni et al. (2013a) and Belloni et al. (2014a) to estimation of ATE when the treatment is multivalued and exogenous conditional on controls using group penalization for selection. Note that this previous work on treatment effects covers only the exogenous case and does not allow for functional responses which are necessary, for example, for working with distributional or quantile treatment effects.

Our work also contributes to the line of research on obtaining -consistent and asymptotically normal estimates for low-dimensional components within traditional semiparametric frameworks as in the important work by Bickel (1982), Robinson (1988), Newey (1990), van der Vaart (1991), Andrews (1994b), Newey (1994), Ai and Chen (2003, 2012), and Chen et al. (2003). The major difference is that we allow for the use of modern high-dimensional methods, a.k.a. machine learning methods, for modeling and fitting the non-parametric (or high-dimensional) components of the model. In contrast to the former literature, we expressly allow for data-driven choice of the approximating model for the high-dimensional component, which addresses a crucial problem that arises in empirical work. Moreover, recent methods based on -penalization, upon which we focus in this paper, allow for much more flexible modeling of the non-parametric/high-dimensional parts of the model.Our general theory in Section 5 also allows, in principle, for a wide variety of both traditional and machine learning methods.

The paper also generates a number of new results on sparse estimation with functional response data. These results are of independent interest in themselves, and they build upon the work of Belloni and Chernozhukov (2011) who provided rates of convergence for variable selection when one is interested in estimating the quantile regression process with exogenous variables. More generally, this theoretical work complements and extends the rapidly growing set of results for estimation methods; see, for example, Frank and Friedman (1993); Tibshirani (1996); Fan and Li (2001); Zou (2006); Cand`es and Tao (2007); van de Geer (2008); Huang et al. (2008); Bickel et al. (2009); Meinshausen and Yu (2009); Bach (2010); Huang et al. (2010); Belloni and Chernozhukov (2011); Kato (2011); Belloni et al. (2012); Belloni and Chernozhukov (2013); Belloni et al. (2013b); Belloni et al. (2013c); Caner and Zhang (2014); and the references therein.

Plan of the Paper. Section 2 introduces the structural parameters for policy evaluation and relates these parameters to reduced form functions. Section 3 describes a three step procedure to estimate and make inference on the structural parameters and functionals of these parameters, and Section 4 provides asymptotic theory in the treatment effects setting. Section 5 generalizes the setting and results to moment-condition problems with a continuum of structural parameters and a continuum of reduced form functions. Section 6 derives general asymptotic theory for the Lasso and post-Lasso estimators for functional response data used in the estimation of the reduced form functions. Section 7 presents the empirical application. We provide notation, proofs of key results, and details about implementation of the methods in the empirical example in Appendices A–E. An on-line Supplementary Appendix provides all remaining proofs, additional technical material, and results from a small Monte Carlo simulation (Belloni et al., 2015).

2.1. Observables and Reduced Form Parameters. The observed random variables consist of ((). The outcome variable of interest is indexed by . We give examples of the index u below. The variable is a binary indicator of the receipt of a treatment or participation in a program. It will typically be treated as endogenous; that is, we will typically view the treatment as assigned non-randomly with respect to the outcome. The instrumental variable is a binary indicator, such as an offer of participation, that is assumed to be randomly assigned conditional on the observable covariates X with support For example, we argue that 401(k) eligibility can be considered exogenous only after conditioning on income and other individual characteristics in the empirical application. The notions of exogeneity and endogeneity we employ are standard and thus omitted.

The indexing of the outcome is useful to analyze functional data. For example, could represent an outcome falling short of a threshold, namely ), in the context of distributional analysis; could be a height indexed by age u in growth charts analysis; or be a health outcome indexed by a dosage u in dosage response studies. Our framework is tailored for such functional response data. The special case with no index is included by simply considering U to be a singleton set.

We make use of two key types of reduced form parameters for estimating the structural parameters of interest – (local) treatment effects and related quantities. These reduced form parameters are defined as

where The function , the support of the vector (Z, X), to the real line R and is defined as

We use V to denote a target variable whose identity may change depending on the context such as ) is the indicator function.

All the structural parameters we consider are smooth functionals of these reduced-form parameters. In our approach to estimating treatment effects, we estimate the key reduced form parameter ) using modern methods to deal with high-dimensional data coupled with orthogonal estimating equations. The orthogonality property allows us to deal with the “non-regular” nature of penalized and post-selection estimators which do not admit linearizations except under very restrictive conditions. The use of regularization by model selection or penalization is in turn motivated by the desire to accommodate high-dimensional data.

2.2. Target Structural Parameters – Local Treatment Effects. The reduced form parameters defined in (2.1) are key because the structural parameters of interest are functionals of these

elementary objects. The local average structural function (LASF) defined as

underlies the formation of many commonly used treatment effects. Under standard assumptions, the LASF identifies average potential outcomes for the group of compliers, individuals whose treatment status may be influenced by variation in the instrument, in the treated and non-treated states; see, e.g. Abadie (2002; 2003). The local average treatment effect (LATE) of Imbens and Angrist (1994) corresponds to the difference of the two values of the LASF:

The term local designates that this parameter does not measure the effect on the entire population but rather measures the effect on the subpopulation of compliers.

When there is no endogeneity, formally when , the LASF and LATE become the average structural function (ASF) and average treatment effect (ATE) on the entire population. Thus, our results cover this situation as a special case where the ASF and ATE simplify to

We also note that the impact of the instrument Z itself may be of interest since Z often encodes an offer of participation in a program. In this case, the parameters of interest are again simply the reduced form parameters

Thus, the LASF and LATE are primary targets of interest in this paper, and the ASF and ATE are subsumed as special cases.

2.2.1. in (2.3) and (2.4) provides the conventional LASF and LATE. An important generalization arises by letting be the indicator of the outcome of interest falling below a threshold . In this case, the family of effects

describe the local distribution treatment effects (LDTE). Similarly, we can look at the quantile left-inverse transform of the curve

and examine the family of local quantile treatment effects (LQTE):

The LQTE identify the differences of quantiles between the distribution of potential outcomes in the treated and non-treated states for compliers.

2.3. Target Structural Parameters – Local Treatment Effects on the Treated. We may also be interested in local treatment effects on the treated. The key object in defining these effects is the local average structural function on the treated (LASF-T) which is defined by its two values:

The LASF-T identifies average potential outcomes for the group of treated compliers in the treated and non-treated states under standard assumptions. The local average treatment effect on the treated (LATE-T) introduced in Hong and Nekipelov (2010) and Fr¨olich and Melly (2013) is the difference of two values of the LASF-T:

The LATE-T may be of interest because it measures the average treatment effect for treated compliers, namely the subgroup of compliers that actually receive the treatment.

When the treatment is assigned randomly given controls so we can take D = Z, the LASF-T and LATE-T become the average structural function on the treated (ASF-T) and average treatment effect on the treated (ATE-T). In this special case, the ASF-T and ATE-T simplify to

and we can use our results to provide estimation and inference methods for these quantities.

2.3.1. Local Distribution and Quantile Treatment Effects on the Treated. Local distribution treatment effects on the treated (LDTE-T) and local quantile treatment effects on the treated (LQTE-T) can also be defined. As in Section 2.2.1, we let ) be the indicator of the outcome of interest falling below a threshold u. The family of treatment effects

then describes the LDTE-T. We can also use the quantile left-inverse transform of the curve and define the LQTE-T:

Under conditional exogeneity LQTE and LQTE-T reduce to the quantile treatment effects (QTE) and quantile treatment effects on the treated (QTE-T) (Koenker, 2005, Chap. 2).

The key objects used to define the structural parameters in Section 2 are the expectations

where denotes a variable whose identity will change with the context. Specifically, we shall vary V over the set

It is clear that ) will play an important role in estimating ). A related function that will also play an important role in forming a robust estimation strategy is the propensity score

We will denote other potential values for the functions by the parameters g and m, respectively. We can then estimate ) by estimating using high-dimensional modeling and estimation methods.

In the rest of this section, we describe the estimation of the reduced-form and structural parameters. The estimation method consists of 3 steps:

1) Estimate the predictive relationships using high-dimensional nonparametric methods with model selection.

2) Estimate the reduced form parameters using orthogonal estimating equations to immunize the reduced form estimators to imperfect model selection in the first step.

3) Estimate the structural parameters and effects via the plug-in rule.

3.1. In this section, we discuss estimation of the conditional expectation functions Since these functions are unknown and potentially complicated, we use a generalized linear combination of a large number of control terms

to approximate . Specifically, we use

In these equations, ) are approximation errors, and the functions Λand Λ) are generalized linear approximations to the target functions The functions Λare taken to be known link functions Λ. The most common example is the linear link Λ(u) = u. When the response variable is binary, we may also use the logistic link Λ() and its complement 1 ) or the probit link Λ(and its complement 1 For clarity, we use links from the finite set where Id is the identity (linear) link.

As discussed in the Introduction, the dictionary of controls, denoted by f(X), can be “rich” in the sense that its dimension may be large relative to the sample size. Specifically, our results require only that log ) along with other technical conditions. We also note that the functions f forming the dictionary can depend on n, but we suppress this dependence.

Having very many controls f(X) creates a challenge for estimation and inference. A useful condition that makes it possible to perform constructive estimation and inference in such cases is termed approximate sparsity or simply sparsity. Sparsity imposes that there exist approximations of the form given in (3.5)-(3.7) that require only a small number of non-zero coefficients to render the approximation errors small relative to estimation error. More formally, sparsity relies on two conditions. First, there must exist such that, for all

where is the number of non-zero components of vector x and all other norms we use are defined in Appendix A. That is, there are at most components of f(Z, X) and f(X) with nonzero coefficient in the approximations to Second, the sparsity condition requires that the size of the resulting approximation errors is small compared to the conjectured size of the estimation error; namely, for all

Note that the size of the approximating model can grow with n just as in standard series estimation, subject to the rate condition

These conditions ensure that the functions are estimable at a ) rate and are used to derive asymptotic normality results for the structural and reduced-form parameter estimators. They could be relaxed through the use of sample splitting methods as in Belloni et al. (2012).

The high-dimensional-sparse-model framework outlined above extends the standard framework in the program evaluation literature which assumes both that the identities of the relevant controls are known and that the number of such controls s is small relative to the sample size.we assume that there are many, p, potential controls of which at most s controls suffice to achieve a desirable approximation to the unknown functions ; and we allow the identity and number of these controls to be unknown. Relying on this assumed sparsity, we use selection methods to choose approximately the right set of controls.

Current estimation methods that exploit approximate sparsity employ different types of regularization aimed at producing estimators that theoretically perform well in high-dimensional settings while remaining computationally tractable. Many widely used methods are based on penalization. The Lasso method is one such commonly used approach that adds a penalty for the weighted sum of the absolute values of the model parameters to the usual objective function of an M-estimator. A related approach is the Post-Lasso method which performs re-estimation of the model after selection of variables by Lasso. These methods are discussed at length in recent papers and review articles; see, for example, Belloni et al. (2013a).

In the following, we outline the general features of the Lasso and Post-Lasso methods focusing on estimation of . Given the data ( ˜, the Lasso estimator

where ) is a diagonal matrix of data-dependent penalty loadings, M(y, t) = (2 in the case of linear regression, and Λin the case of binary regression. The penalty level, , and loadings, are selected to guarantee good theoretical properties of the method. We provide further discussion of these methods for estimation of a continuum of functions in Section 6, and we specify detailed implementation algorithms used in the empirical example in Appendix F. A key consideration in this paper is that the penalty level needs to be set to account for the fact that we will be simultaneously estimating potentially a continuum of Lasso regressions since our V varies over the list varying over the index set U.

The Post-Lasso method uses solely as a model selection device. Specifically, it makes use of the labels of the regressors with non-zero estimated coefficients, The Post-Lasso estimator is then a solution to

A main contribution of this paper is establishing that the estimator the regression function , achieves the near oracle rate of convergenceand maintains desirable theoretic properties while allowing for a continuum of response variables.

Estimation of proceeds similarly. The Lasso estimator and Post-Lasso estimator ˜are defined analogously to using the data ( ˜. The estimator , also achieves the near oracle rate of convergenceand has other good theoretic properties. The estimator of is then formed as 1

3.2. Estimation of the key quantities ) will make heavy use of orthogonal moment functions as defined in (1.2). These moment functions are closely tied to efficient influence functions, where effi-ciency is in the sense of locally minimax semi-parametric efficiency. The use of these functions will deliver robustness with respect to the non-regularity of the post-selection and penalized estimators needed to manage high-dimensional data. The use of these functions also automatically delivers semi-parametric efficiency for estimating and performing inference on the reduced-form parameters and their smooth transformations – the structural parameters.

The efficient influence function and orthogonal moment function for given respectively by

This efficient influence function was derived by Hahn (1998); it has recently been used by Cattaneo (2010) in the series context (with ) and Rothe and Firpo (2013) in the kernel context. The efficient influence function and the moment function for are trivially given by

We then define estimators of the reduced-form parameters ) as solutions to the equations

where are constructed as in Section 3.1. We apply this procedure to each variable name and obtain the estimator

The estimator and the parameter are vectors in with dimension

where is a rich set of data generating processes P which includes cases where perfect model selection is impossible theoretically. The notation “uniformly in ” is defined formally in Appendix A and can be read as “is approximately distributed as uniformly in .” This usage corresponds to the usual notion of asymptotic distribution extended to handle uniformity in P.

We then stack all the reduced form estimators and parameters over

giving rise to the empirical reduced-form process and the reduced-form function-valued parameter . We establish that ) is asymptotically Gaussian: In

where denotes the P-Brownian bridge (van der Vaart and Wellner, 1996, p. 81–82). This result contains (3.17) as a special case and again allows to be a “rich” set of data generating processes P that includes cases where perfect model selection is impossible theoretically. Importantly, this result verifies that the functional central limit theorem applies to the reduced-form estimators in the presence of possible model selection mistakes.

Since some of our objects of interest are complicated, inference can be facilitated by a multiplier bootstrap method as in Gin´e and Zinn (1984). We define , a bootstrap draw of

Here (are i.i.d. copies of which are independently distributed from the data (whose distribution does not depend on P. We also impose that

Examples of include (a) is a standard exponential random variable, (b) where N is a standard normal random variable, and (c) are mutually independent standard normal random variables.The choices of (a), (b), and (c) correspond respectively to the Bayesian bootstrap (e.g., Hahn (1997) and Chamberlain and Imbens (2003)), the Gaussian multiplier method (e.g, Gin´e and Zinn (1984) and van der Vaart and Wellner (1996, Chap. 3.6)), and the wild bootstrap method (Mammen (1993)).in (3.19) is an estimator of the influence function defined via the plug-in rule:

Note that this bootstrap is computationally efficient since it does not involve recomputing the influence functions Each new draw of (generates a new draw of holding the data and the estimates of the influence functions fixed. This method simply amounts to resampling the first-order approximations to the estimators. Here we build upon prior uses of this or similar methods in low-dimensional settings such as Hansen (1996) and Kline and Santos (2012).

where denotes weak convergence of the bootstrap law in probability, as defined in Appendix B.

3.3. Third Step: Robust Estimation of the Structural Parameters. All structural parameters we consider take the form of smooth transformations of the reduced-form parameters:

The structural parameters may themselves carry an index that can be different from u; for example, the LQTE is indexed by a quantile index 1). This formulation includes as special cases all the structural functions of Section 2. We estimate these quantities by the plug-in rule. We establish the asymptotic behavior of these estimators and the validity of the bootstrap as a corollary from the results outlined in Section 3.2 and the functional delta method (extended to handle uniformity in P).

For the application of the functional delta method, we require that the functional be Hadamard differentiable is a set that contains the true values , tangentially to a subset that contains the realizations of with derivative map We define the estimators of the structural parameters and their bootstrap versions via the plug-in rule as

We establish that these estimators are asymptotically Gaussian

and that the bootstrap consistently estimates their large sample distribution:

These results can be used to construct simultaneous confidence bands and test functional hypotheses on ∆ using the methods described for example in Chernozhukov and Fern´andez-Val (2005) and Chernozhukov et al. (2013).

Consider fixed sequences of numbers 0, at a speed at most polynomial in n (for example, , and positive constants These sequences and constants will not vary with P. The probability P can vary in the set of probability measures, termed “data-generating processes”, where is typically a set that is weakly increasing in . Other definitions and notation are collected in Appendix A.

Assumption 4.1 (Basic Assumptions). (i) Consider a random element W with values in a measure space (and law determined by a probability measure The observed data ((i.i.d. copies of a random element (U is a Polish space equipped with its Borel sigma-field and (generated via a measurable transform t(W, u) of W and u, namely the map

is measurable, and the map can possibly depend on P. Let

where obeys the uniform continuity property:

where the second supremum in the first expression is taken over is a totally bounded metric space equipped with a semi-metric . The uniform covering entropy of the set , viewed as a collection of maps (

for all , with the supremum taken over all finitely discrete probability measures . (iii) For each , the conditional probability of Z = 1 given X is bounded away from zero or one, namely -a.s., the instrument Z has a non-trivial impact on and the regression function is bounded,

Assumption 4.1 is stated to deal with the measurability issues associated with functional response data. This assumption also implies that the set of functions (

is P-Donsker uniformly in P. That is, it implies

with denoting the P-Brownian bridge (van der Vaart and Wellner, 1996, p. 81–82) and with having bounded, uniformly continuous paths uniformly in

We work with the sequence of constants defined prior to Assumption 4.1.

Assumption 4.2 (Approximate Sparsity). Under each and for each , uniformly for all : (i) The approximations (3.5)-(3.7) hold with the link functions Λlonging to the set L, the sparsity condition holding, the approximation errors satisfying , and the sparsity index s and the number of terms p in the vector . (ii) There are estimators ¯such that, with probability no less than 1 , the estimation errors satisfy ; the estimators are sparse such that ; and the empirical and population norms induced by the Gram matrix formed by (are equivalent on sparse subsets, sup. (iii) The following boundedness conditions hold:

Comment 4.1. Assumption 4.2 imposes simple intermediate-level conditions which encode both the approximate sparsity of the models as well as some reasonable behavior of the sparse estimators of . These conditions significantly extend and generalize the conditions employed in the literature on adaptive estimation using series methods. The boundedness conditions are made to simplify arguments, and they could be removed at the cost of more complicated proofs and more stringent side conditions. Sufficient conditions for the equivalence between empirical and population norms and primitive examples of functions admitting sparse approximations are given in Belloni et al. (2014a). We provide primitive conditions for Lasso estimators to satisfy the bounds above while addressing the problem of estimating continua of approximately sparse nuisance functions in Section 6. We expect that other sparsity-based estimators, such as the Dantzig selector or adaptive Lasso, could be used in the present context as well.

Under the stated assumptions, the empirical reduced form process ) defined by (3.16) obeys the following relations. We recall definitions of convergence uniformly in Appendix A.

Theorem 4.1 (Uniform Gaussianity of the Reduced-Form Parameter Process). Under Assumptions 4.1 and 4.2, the reduced-form empirical process admits a linearization; namely,

The process is asymptotically Gaussian, namely

where is defined in (4.2) and its paths obey the property (4.3).

Another main result of this section shows that the bootstrap law of the process

Theorem 4.2 (Validity of Multiplier Bootstrap for Inference on Reduced-Form Parameters). Under Assumptions 4.1 and 4.2, the bootstrap law consistently approximates the large sample law uniformly in

Next we consider inference on the structural functionals ∆ defined in (3.22). We derive the large sample distribution of the estimator ∆ in (3.23), and show that the multiplier bootstrap law of in (3.23) provides a consistent approximation to that distribution. We rely on the functional delta method in our derivations, which we modify to handle uniformity with respect to the underlying d.g.p. P. Our argument relies on the following assumption on the structural functionals.

Assumption 4.3 (Uniform Hadamard Differentiability of Structural Functionals). Suppose that for each , a compact metric space. Suppose , a functional of interest mapping , is Hadamard differentiable in tangentially to uniformly in , with the linear derivative map such that the mapping (is continuous.

The definition of uniform Hadamard differentiability is given in Definition B.1 of Appendix B. Assumption 4.3 holds for all examples of structural parameters listed in Section 2.

The following corollary gives the large sample law of ∆), the properly normalized structural estimator. It also shows that the bootstrap law of computed conditionally on the data, approaches the large sample law ∆). It follows from the previous theorems as well as from a more general result contained in Theorem 5.3.

Corollary 4.1 (Limit Theory and Validity of Multiplier Bootstrap for Smooth Structural Functionals). Under Assumptions 4.1, 4.2, and 4.3,

where is a zero mean tight Gaussian process, for each . Moreover,

In this section, we consider a general moment condition framework, where possibly a continuum of target parameters is of interest and we use modern machine learning methods, with Lasso-type methods being a lead example, to estimate a continuum of high-dimensional nuisance functions. This setting covers a rich variety of modern moment-condition problems in econometrics including the treatment effects problem. We establish a functional central limit theorem for the estimators of the continuum of target parameters that holds uniformly in includes a wide range of data-generating processes with well-approximable continuums of nuisance functions. We also derive a functional central limit theorem for the multiplier bootstrap that resamples the first order approximations to the standardized estimators of the continuum of target parameters and establish its uniform validity. Moreover, we establish the uniform validity of the functional delta method and the functional delta method for the multiplier bootstrap for smooth functionals of the continuum of target parameters using an appropriate strengthening of Hadamard differentiability.

5.1. Setting. We are interested in function-valued target parameters indexed by We denote the true value of the target parameter by

We assume that for each the true value is identified as the solution to the following moment condition:

where is a random vector that takes values in a Borel set and contains as a subcomponent the vector taking values in a Borel set , the moment function

is a Borel measurable map, and the function

is another Borel measurable map that denotes the possibly infinite-dimensional nuisance parameter. The sets ) are assumed to be convex for each . Finite-dimensional nuisance parameters that do not depend on are treated as part of

We assume that the continuum of nuisance functions (is well-approximable and can be well estimated by the modern generation of statistical and machine learning methods. In particular, our regularity conditions allow for approximately sparse nuisance functions, which can be modeled and estimated using methods such as Lasso and Post-Lasso. We let estimator of , which we assume obeys the conditions in Assumption 5.3. The estimator is constructed as any approximate -solution in Θto a sample analog of the moment condition (5.1), i.e.,

Comment 5.1 (Handling Over-identified Cases). We do not analyze over-identified cases explicitly, but it is helpful to note that they can be handled within the current framework. Let )) be the original over-identifying moment function. Let ) denote the point- wise optimal matrix of linear combinations of the moments, so that the final moment function )) has the same dimension as (vec(; that is, we simply treat as part of the nuisance function estimated. We do not analyze the preliminary estimation of in the present paper in order to maintain the focus on exactly identified cases as in Section 4.

5.2. The Neyman Orthogonality or Immunization Condition. A key condition needed for regular estimation of is an orthogonality or immunization condition. The simplest to explain, yet strongest, form of this condition can be expressed as follows:

subject to additional technical conditions such as continuity (5.6) and dominance (5.7) stated below, where we use the symbol to abbreviate . This condition holds in the previous setting of inference on treatment effects after interchanging the order of the derivative and expectation. The formulation here also covers certain non-smooth cases such as structural and instrumental quantile regression problems.

In the formal development, we use a more general form of the orthogonality condition.

Definition 5.1 (Neyman Orthogonality for Moment Condition Models, General Form). For each , suppose that (5.1)–(5.3) hold. Consider , a set of measurable functions . Suppose also that the set is a convex subset of . We say that obeys a general form of orthogonality with respect to uniformly in if the following conditions hold: For each , the derivative

and obeys the orthogonality condition:

The orthogonality condition (5.8) reduces to (5.5) when can span all measurable functions but is more general otherwise.

Comment 5.2 (An alternative formulation of the Neyman orthogonality condition). A slightly more general, though less primitive definition of the orthogonality condition is as follows. For each , suppose that (5.1)- (5.3) hold. Consider , a set of measurable functions the set ) is a convex subset of . We say that obeys a general form of orthogonality with respect to uniformly in , if the following conditions hold: The Gateaux derivative map

exists for all and vanishes at t = 0 – namely,

Definition 5.1 implies this definition by the mean-value expansion and the dominated convergence theorem.

Comment 5.3 (Orthogonalization typically expands the number of nuisance parameters). It is important to use a moment function that satisfies the orthogonality property given in (5.8); see examples given below. Generally, if we have a moment function ˜which identifies but does not have this property, we can construct a moment function that identifies the required orthogonality property by projecting the original function ˜onto the orthocomplement of the tangent space for the original set of nuisance functions ; see, for example, van der Vaart and Wellner (1996), van der Vaart (1998, Chap. 25), Kosorok (2008), Belloni et al. (2013b), and Belloni et al. (2014a). This projection creates the semi-parametrically efficient score function. There are other ways to create orthogonal nuisance functions, as illustrated by the second example below.

Note that the projection typically depends on P, which gives rise to additional nuisance parameters , which are then incorporated together with the original nuisance parameters into the new parameter ). Note that this is a feature of all of the examples we consider. For ex- ample, the orthogonal moment functions in the exogenous case of the treatment effects framework depend on both the regression function and the propensity score function. This point is clarified further by considering the classical linear model as demonstrated in the next remark.

Example 1 (Neyman Orthogonal Equations for Linear Regression). To illustrate the orthogonality condition in the simplest possible setting, let us consider the linear model:

where D is the treatment and X are the controls of high dimension . Call the first equation the regression equation, and the second equation the propensity score equation. The orthogonal moment condition that identifies the projection coefficient is the Frisch-Waugh-Lovell partialling out interpretation of

where U is the population residual left after projecting out the controls X from the outcome, i.e. is the population residual left after projecting out controls from the treatment as defined in the propensity score equation. The high-dimensional nuisance function is , with true value denoted by moment function

has the required orthogonality property (5.8), since by the law of iterated expectations and some simple algebra:

for ) is the semi-parametrically efficient score for The resulting estimator of consistent and asymptotically normal, uniformly within a class of approximately sparse models as follows from the general results of this section, and is also semi-parametrically efficient. See also Belloni et al. (2014a) which deals with the partially linear model in detail and thus covers this linear example as a special case.

Note that the orthogonal moment function contains two nuisance functions – the regression function and the propensity score – . We could also identify through non-orthogonal moment conditions containing single nuisance functions:

The first moment condition corresponds to the regression method, while the second to the so-called covariate balancing method. Importantly, the use of these non-orthogonal moment conditions generally does not produce an estimator for -consistent and asymptotically normal uniformly in the class of approximately sparse models. This failure occurs because we are forced to use highly non-regular estimators to estimate the nuisance functions setting. In fact, this failure would also occur with a low number of controls, including having only p = 1, whenever selection procedures that exclude irrelevant variables with very high probability are used to estimate the regression parameter or the propensity score parameter discussion and documentation of this failure, see Leeb and P¨otscher (2008a; 2008b); P¨otscher (2009); and Belloni, Chernozhukov, and Hansen (2013a; 2014a). By contrast, constructing orthogonal moment conditions – involving the projection of both the outcome and the treatment onto the controls and thereby combining the regression and covariate balancing methods – makes it possible to achieve consistency and asymptotic normality uniformly within a class of approximately sparse models.

Example 2 (Neyman Orthogonal Equations for a Class of Conditional Moment Problems). Next, consider the conditional moment restrictions framework studied by Chamberlain (1992):

where X and W are random vectors with X being a sub-vector of is a finite-dimensional parameter whose true value is of interest, g is a functional nuisance parameter mapping the support of X into a convex set whose true value is is a known function with values in

Here we would like to build a score function () for estimating value of parameter is a new nuisance parameter with true value that obeys the strong form of the orthogonality condition (5.5) and thus also its weak form (5.8). To this end, let ] be a function mapping matrix of its derivatives. We will set where is a function mapping the support of X into the space of the function mapping the support of X into the space of . Define the true value

where matrix of measurable transformations of identity matrix, and Πnon-identity matrix with the property:

where Σis the true value of parameter Σ. For example, Π) can be chosen to be the idempotent matrix:

Then an orthogonal score for the problem above can be constructed as

It is straightforward to check that under mild regularity conditions that the score function E)) and also obeys the orthogonality condition:

Furthermore, by setting

and using Π) suggested above, we obtain the efficient score that yields an estimator of achieving the semi-parametric efficiency bound provided in Chamberlain (1992).

Here we would like to note that an analogous, though more involved, construction can be provided for the more general class of problems considered in Ai and Chen (2003) where the nuisance functions depend on the endogenous variables.

5.3. Regularity Conditions and Results. In what follows, we shall denote by some positive constants. For a positive integer d, [d] denotes the set {1, . . . , d}. We shall impose the following regularity conditions.

Assumption 5.1 (Moment condition problem). Consider a random element W, taking values in a measure space (, with law determined by a probability measure . The observed data ((i.i.d. copies of a random element (which is generated as a suitably measurable transformation with respect to W and u. Uniformly for all , the following conditions hold: (i) The true parameter value obeys (5.1) and is interior relative to Θ, namely there is a ball of radius centered at contained in Θall is compact. (ii) For Θis twice continuously differentiable a.s. with derivatives obeying the integrability conditions specified in Assumption 5.2. (iii) For all the moment function obeys the orthogonality condition given in Definition 5.1 for the set in Assumption 5.3. (iv) The following identifiability condition holds: 2where the singular values of lie between

The conditions of Assumption 5.1 are mild and standard in moment condition problems. Assumption 5.1(iv) encodes sufficient global and local identifiability to obtain a rate result. The suitably measurable condition, defined in Appendix A, is a mild condition satisfied in most practical cases.

Assumption 5.2 (Entropy and smoothness)is a semi-metric space such that log be some positive constants. Uniformly for all , the following conditions hold: (i) The set of functions , viewed as functions of W is suitably measurable; has an envelope function that is measurable with respect to is a fixed constant; and has a uniform covering entropy obeying sup(ii) For all and

Assumption 5.2 imposes smoothness and integrability conditions on various quantities derived from . It also imposes conditions on the complexity of the relevant function classes.

In what follows, let ∆0 be sequences of constants approaching zero from above at a speed at most polynomial in n (for example,

Assumption 5.3 (Estimation of nuisance functions). The following conditions hold for each and all . The estimated functions with probability at least 1 where is the set of measurable maps

and whose complexity does not grow too quickly in the sense that [is suitably measurable and its uniform covering entropy obeys

where is an envelope for which is measurable with respect to W and satisfies defined in Assumption 5.2. The complexity characteristics obey the growth conditions:

where are defined in Assumption 5.2.

Comment 5.4 (On Rate and Entropy Rate Conditions). Assumption 5.3 imposes conditions on the estimation rate of the nuisance functions and on the complexity of the functions sets that contain the estimators . This condition allows for a wide variety of modern modeling assumptions and regularization methods for function fitting, including both traditional methods and more recent statistical and machine learning methods. Within the approximately sparse framework, the index corresponds to the maximum of the dimension of the approximating models and of the size of the selected models; and . Under other frameworks, these parameters could be different; yet if they are well-behaved, then our results still apply. Thus, these results cover other frameworks, where structured assumptions other than approximate sparsity are used to make the estimation and modeling problem manageable. It is important to point out that the class generally will not be Donsker because its entropy is allowed to increase with n. Allowing for non-Donsker classes is crucial for accommodating modern, high-dimensional estimation methods for the nuisance functions. This feature makes the conditions imposed here very different from the conditions imposed in various classical references on dealing with nonparametrically estimated nuisance functions; see, for example, van der Vaart and Wellner (1996), van der Vaart (1998), Kosorok (2008), and other references listed in the introduction.

Comment 5.5 (Removing Entropy Rate Conditions by Sample-Splitting). We can can set and in Assumption 5.3 if we employ data-splitting. That is, under data-splitting the entropy condition becomes very weak, akin to that in parametric problems, facilitating the application of modern statistical and machine learning methods (e.g. random forest, boosted trees, deep neural nets, and their aggregated and hybrid versions) to estimate the nuisance functions. Thus, with data-splitting Assumption 5.3 only requires that the estimators of nuisance parameters attain sufficiently rapid rates of convergences , in particular ) in smooth problems. Of course in practice we can not verify that these rates hold in a given problem, but the regularity conditions become more plausible with data-splitting than without it. Bickel (1982) employs the idea of data-splitting, namely setting aside a vanishing fraction of the sample to estimate the nuisance parameter, to set up adaptive estimators of the main parameter; see also van der Vaart (1998). This ensures that there is no asymptotic efficiency loss from data-splitting. Another method, which seems more practical, is to use the following cross-fitting approach: (1) split the sample into two equal parts, the auxiliary and main parts; (2) use the auxiliary part to estimate the nuisance parameter and the main part to estimate the target parameter, obtaining one estimator of the target parameter; (3) by reversing the roles of the main and auxiliary parts, obtain another estimator of the target parameter; and (4) average the two estimators of the target parameter to obtain the final estimator. Theorems 5.1 given below yields the properties of the final estimator. We refer to Chernozhukov et al. (2016) for further details, including the result that there is no asymptotic efficiency loss from data-splitting under cross-fitting.

The following theorem is one of the main results of the paper:

Theorem 5.1 (Uniform Functional Central Limit Theorem for a Continuum of Target Parameters in Moment Condition Problems). Under Assumptions 5.1, 5.2, and 5.3, for an

in uniformly in

where the paths of are a.s. uniformly continuous on (

Comment 5.6. It is important to mention here that this result on a continuum of parameters solving a continuum of moment conditions is completely new. The prior approaches dealing with continua of moment conditions with infinite-dimensional nuisance parameters, for example, the ones given in Chernozhukov and Hansen (2006) and Escanciano and Zhu (2013), impose Donsker conditions on the class of functions, following Andrews (1994a), that contain the values of the estimators of these nuisance functions. This approach is precluded in our setting because the resulting class of functions in our case has entropy that grows with the sample size and therefore is not Donsker. Hence, we develop a new approach to establishing the results which exploits the interplay between the rate of growth of entropy, the biases, and the size of the estimation error. In addition, the new approach allows for obtaining results that are uniform in

We can estimate the law of with the bootstrap law of

and is a suitable estimator of The bootstrap law is computed by drawing (conditional on the data.

The following theorem shows that the multiplier bootstrap provides a valid approximation to the large sample law of

Theorem 5.2 (Uniform Validity of Multiplier Bootstrap). Suppose Assumptions 5.1, 5.2, and 5.3 hold, the estimator (obeys equation (5.4), and that the estimator ( following condition: uniformly in with probability 1

We next derive the large sample distribution and validity of the multiplier bootstrap for the estimator ) using the functional delta method. The functional ) is defined as a uniformly Hadamard differentiable transform of . The following result gives the large sample law of properly normalized estimator. It also shows that the bootstrap law of conditionally on the data, is consistent for the large sample law of ) is the bootstrap version of ) is the multiplier bootstrap version of defined via equation (5.16).

Theorem 5.3 (Uniform Limit Theory and Validity of Multiplier Bootstrap for Smooth Suppose that for each is an element of a compact set , a functional of interest mapping , is Hadamard differentiable in tangentially to uniformly in the linear derivative map such that the mapping (is continuous. Then,

where is a zero mean tight Gaussian process, for each . Moreover,

To derive Theorem 5.3, we strengthen the usual notion of Hadamard differentiability to a uniform notion introduced in Definition B.1. Theorems B.3 and B.4 show that this uniform Hadamard differentiability is sufficient to guarantee the validity of the functional delta uniformly in P. These new uniform functional delta method theorems may be of independent interest.

In this section, we provide results for Lasso and Post-Lasso estimators with function-valued outcomes and linear or logistic links. As these results are of interest beyond the context of estimation of nuisance functions for moment condition problems or treatment effects estimation, we present this section in a way that leaves it autonomous with respect to the rest of the paper.

6.1. The generic setting with function-valued outcomes. Consider a data generating process with a functional response variable (and observable covariates X satisfying for each

where is a set of p measurable transformations of the initial controls dimensional vector, is an approximation error, and Λ is a fixed known link function. The notation in this section differs from the rest of the paper with denoting a generic response and a generic vector of covariates to facilitate the application of these results to other contexts. We only consider the linear link function, Λ(t) = t, and the logistic link function, Λ(t) = exp(t)/{1+exp(t)}, in detail.

Considering the logistic link is useful when the functional response is binary, though the linear link can be used in that case as well under some conditions. For example, it is useful for estimating a high-dimensional generalization of the distributional regression models considered in Chernozhukov et al. (2013) where the response variable is the continuum (. Even though we focus on these two cases we note that the principles discussed here apply to many other M-estimators with convex (or approximately convex) criterion functions. In the remainder of the section, we discuss and establish results for -penalized and post-model selection estimators of (that hold uniformly over

Throughout the section, we assume that and that we have n i.i.d. observations from d.g.p.’s where (6.1) holds, , available for estimating (penalty level , and diagonal matrix of penalty loadings we define the Lasso estimator as

where in the case of linear regression, and 1(in the case of the logistic link function for binary response data. For each , the Post-Lasso estimator based on a set of covariates is then defined as

where the set contains supp() and may also contain additional variables deemed as important.We will set ) unless otherwise noted.

The chief departure between the analysis when U is a singleton and the functional response case is that the penalty level needs to be set to control selection errors uniformly over . To do so, we will set so that with high probability

where c > 1 is a fixed constant. When U is a singleton the strategy above is similar to Bickel et al. (2009), Belloni and Chernozhukov (2013), and Belloni et al. (2011), who use an analog of (6.4) to derive the properties of Lasso and Post-Lasso. When U is not a singleton, this strategy was first employed in the context of -penalized quantile regression processes by Belloni and Chernozhukov (2011).

To implement (6.4), we propose setting the penalty level as

where is the dimension of (1) is a confidence level associated with the probability of event (6.4), and c > 1 is a slack constant.When implementing the estimators, we set ), which is theoretically motivated and practically tested in an extensive set of simulation experiments in Belloni et al. (2014a). In addition to the penalty parameter , we also need to construct a penalty loading matrix This loading matrix can be formed according to the following iterative algorithm.

Algorithm 6.1 (Estimation of Penalty Loadings)as defined in (6.5), and choose a constant as an upper bound on the number of iterations. (0) Set k = 0, and initialize . For the linear link function, set . For the logistic link function, set . (1) Compute the Lasso and Post-Lasso estimators, on k > K, stop; otherwise set and go to step (1).

6.2. Properties of a Continuum of Lasso and Post-Lasso: Linear Link. We provide suf-ficient conditions for establishing good performance of the estimators discussed above when the linear link function is used. In the statement of the following assumption, are fixed sequences approaching zero from above at a speed at most polynomial in n (for example, 1] are positive finite constants.

Assumption 6.1. Consider a random element W taking values in a measure space (with law determined by a probability measure . The observed data ((of n i.i.d. copies of random element ((, which is generated as a suitably measurable transformation of W and u. The model (6.1) holds with linear link [0is fixed and U is equipped with the semi-metric . Uniformly for all and , the following conditions hold. (i) The model (6.1) is approximately sparse with sparsity index obeying supand the growth restriction log(set U has uniform covering entropy obeying log , and the collection (are suitably measurable transformations of W and u. (iii) Uniformly over , the moments of the model are boundedly heteroscedastic, namely a.s., and max(iv) For a fixed and a sequence the dictionary functions, approximation errors, and empirical errors obey the following regularity conditions: (a) (b) With probability 1 E. (c) With probability 1 the empirical minimum and maximum sparse eigenvalues are bounded from zero and above, namely

Assumption 6.1 is only a set of sufficient conditions. The finite sample results in the Supplementary Appendix allow for more general conditions (for example, can grow with the sample size). We verify that the more technical conditions in Assumption 6.1(iv)(b) hold in a variety of cases, see Lemma I.2 in Appendix I in the Supplementary Appendix. Under Assumption 6.1, we establish results on the performance of the estimators (6.2) and (6.3) for the linear link function case that hold uniformly over

Theorem 6.1 (Rates and Sparsity for Functional Responses under Linear Link). Under Assumption 6.1 and setting the penalty and loadings as in Algorithm 6.1, for all n large enough, uniformly for all probability 1, for some constant ¯C, the Lasso estimator is uniformly sparse, sup, and the following performance bounds hold:

For all n large enough, uniformly for all probability 1 , the Post-Lasso estimator corresponding to

We note that the performance bounds are exactly of the type used in Assumption 4.2 (see also Assumption H.1 in the Supplementary Appendix). Indeed, under the condition , the rate of convergence established in Theorem 6.1 yields

6.3. Properties of Lasso and Post-Lasso Estimators: Logistic Link. We provide sufficient conditions to state results on the performance of the estimators discussed above for the logistic link function. Consider the fixed sequences 0 approaching zero from above at a speed at most polynomial in , and the positive finite constants

The following result characterizes the performance of the estimators (6.2) and (6.3) for the logistic link function case under Assumption 6.2.

Theorem 6.2 (Rates and Sparsity for Functional Response under Logistic Link). Under Assumption 6.2 and setting the penalty and loadings as in Algorithm 6.1, for all n large enough, uniformly for all probability 1, the following performance bounds hold for some constant ¯C:

and the estimator is uniformly sparse: suplarge enough, uniformly for all probability 1 , the Post-Lasso estimator corresponding to

Comment 6.1. The performance bounds derived in Theorem 6.2 satisfy the conditions of Assumption 4.2 (see also Assumption H.1 in the Supplementary Material). Moreover, since the link function is 1-Lipschitz in the logistic case and the approximation errors are assumed to be small, the results above establish the same rates of convergence for estimators of the conditional probabilities; for example,

As a practical illustration of the methods developed in this paper, we consider estimation of the effect of 401(k) eligibility and participation on accumulated assets as in Abadie (2003) and Chernozhukov and Hansen (2004). Our goal here is to illustrate the estimation results and inference statements and to make the following points that underscore our theoretical findings: 1) In a low-dimensional setting, where the number of controls is low and therefore there is no need for selection, our robust post-selection inference methods perform well. That is, the results of our methods agree with the results of standard methods that do not employ any selection. 2) In a high-dimensional setting, where there are (moderately) many controls, our post-selection inference methods perform well, producing well-behaved estimates and confidence intervals compared to the erratic estimates and confidence intervals produced by standard methods that do not employ selection as a means of regularization. 3) Finally, in a very high-dimensional setting, where the number of controls is comparable to the sample size, the standard methods break down completely, while our methods still produce well-behaved estimates and confidence intervals. These findings are in line with our theoretical results about uniform validity of our inference methods.

The key problem in determining the effect of participation in 401(k) plans on accumulated assets is saver heterogeneity coupled with the fact that the decision to enroll in a 401(k) is non-random. It is generally recognized that some people have a higher preference for saving than others. It also seems likely that those individuals with high unobserved preference for saving would be most likely to choose to participate in tax-advantaged retirement savings plans and would tend to have otherwise high amounts of accumulated assets. The presence of unobserved savings preferences with these properties then implies that conventional estimates that do not account for saver heterogeneity and endogeneity of participation will be biased upward, tending to overstate the savings effects of 401(k) participation.

To overcome the endogeneity of 401(k) participation, Abadie (2003) and Chernozhukov and Hansen (2004) adopt the strategy detailed in Poterba, Venti, and Wise (1994; 1995; 1996; 2001) and Benjamin (2003), who used data from the 1991 Survey of Income and Program Participation and argue that eligibility for enrolling in a 401(k) plan in this data can be taken as exogenous after conditioning on a few observables of which the most important for their argument is income. The basic idea of their argument is that, at least around the time 401(k)’s initially became available, people were unlikely to be basing their employment decisions on whether an employer offered a 401(k) but would instead focus on income. Thus, eligibility for a 401(k) could be taken as exogenous conditional on income, and the causal effect of 401(k) eligibility could be directly estimated by appropriate comparison across eligible and ineligible individuals.Abadie (2003), Chernozhukov and Hansen (2004), and Ogburn et al. (2015) use this argument for the exogeneity of eligibility conditional on controls to argue that 401(k) eligibility provides a valid instrument for 401(k) participation and employ IV methods to estimate the effect of 401(k) participation on accumulated assets.

As a complement to the work cited above, we estimate various treatment effects of 401(k) participation on financial wealth using high-dimensional methods. A key component of the argument underlying the exogeneity of 401(k) eligibility is that eligibility may only be taken as exogenous after conditioning on income. Both Abadie (2003) and Chernozhukov and Hansen (2004) adopt this argument but control only for a small number of terms. One might wonder whether the small number of terms considered is sufficient to adequately control for income and other related confounds. At the same time, the power to learn anything about the effect of 401(k) participation decreases as one controls more flexibly for confounds. The methods developed in this paper offer one resolution to this tension by allowing us to consider a very broad set of controls and functional forms under the assumption that among the set of variables we consider there is a relatively low-dimensional set that adequately captures the effect of confounds. This approach is more general than that pursued in previous research which implicitly assumes that confounding effects can adequately be controlled for by a small number of variables chosen ex ante by the researcher.

We use the same data as Chernozhukov and Hansen (2004). The data consist of 9,915 observations at the household level drawn from the 1991 SIPP. We use net financial assets as the outcome variable, Y , in our analysis. Our treatment variable, D, is an indicator for having positive 401(k) balances; and our instrument, Z, is an indicator for being eligible to enroll in a 401(k) plan. The vector of raw covariates, X, consists of age, income, family size, years of education, a married indicator, a two-earner status indicator, a defined benefit pension status indicator, an IRA participation indicator, and a home ownership indicator. Further details can be found in Chernozhukov and Hansen (2004).

We present detailed results for three different sets of controls f(X). The first specification uses indicators of marital status, two-earner status, defined benefit pension status, IRA participation status, and home ownership status, second order polynomials in family size and education, a third order polynomial in age, and a quadratic spline in income with six break pointsSpline specification). The second specification augments the Quadratic Spline specification by interacting all the non-income variables with each term in the income spline (Quadratic Spline Plus Interactions specification). The final specification forms a larger set of potential controls by starting with all of the variables from the Quadratic Spline specification and forming all two-way interactions between all of the non-income variables. The set of main effects and interactions of all non-income variables is then fully interacted with all of the income terms (Quadratic Spline Plus Many Interactions specification).The dimensions of the set of controls are thus 35, 311, and 1756 for the Quadratic Spline, Quadratic Spline Plus Interactions, and Quadratic Spline Plus Many Interactions specification, respectively. For methods that do not use variable selection, we use 32, 272, and 1526 variables resulting from removing terms that are perfectly collinear. We refer to the specification without interactions as low-p, to the specification with only income interactions as high-p, and to the specification with all two-way interactions further interacted with income as very-high-p.

We report a variety of results for each specification. Under the maintained assumption that 401(k) eligibility may be taken as exogenous after controlling for the variables defined in the preceding paragraph, we can use the methods of this paper to estimate intention to treat effects of 401(k) eligibility by setting 401(k) eligibility as D = Z. We report the estimated average intention to treat and average intention to treat on the treated as the ATE and ATE-T, and we report estimates of quantile intention to treat and quantile intention to treat on the treated effects as QTE and QTE-T. We also directly apply the results of this paper to estimate effects of 401(k) participation, reporting estimates of the LATE, LATE-T, LQTE, and LQTE-T for each specifi-cation.For comparison, we also report estimates of the eligibility effect from the linear model without selection and with selection using the approach of Belloni et al. (2014a) and estimates of the participation effect from linear instrumental variables estimation without selection and with selection as in Chernozhukov et al. (2015a).

Estimation of all these treatment effects depends on first-stage estimates of reduced form functions as detailed in Section 3. We estimate reduced form functions where the outcome is continuous using ordinary least squares when no model selection is used or Post-Lasso when selection is used. We estimate reduced form functions where the outcome is binary by logistic regression when no model selection is used or Post--penalized logistic regression when selection is used. We only report selection-based estimates in the very-high-We refer to Appendix F for detailed discussion of implementing our approach in this example.

Estimates of the ATE, ATE-T, LATE and LATE-T as well as the coefficient on 401(k) eligibility from the linear model and coefficient on 401(k) participation in the linear IV model are given in Table 1. In this table, we provide point estimates for each of the three sets of controls with and without variable selection. We report conventional heteroscedasticity consistent standard error estimates for the linear model and linear IV coefficient. For the ATE, ATE-T, LATE, and LATE-T, we report both analytic and multiplier bootstrap standard errors. The bootstrap standard errors are based on 500 bootstrap replications with Mammen (1993) weights as multipliers.

Looking first at the two sets of standard error estimates for the average treatment effect estimates, we see that the bootstrap and analytic standard errors are quite similar and that one would not draw substantively different conclusions from using one versus the other. We also see that estimates of the effect of 401(k) eligibility using the linear model and estimates of the effect of 401(k) participation using the linear IV model are broadly consistent with each other across all specifications and regardless of whether or not variable selection is done. We also have that the estimates of the ATE, ATE-T, LATE, and LATE-T are very similar regardless of whether selection is used in the low-p Quadratic Spline specification. The ATE and ATE-T both indicate a positive and significant average effect of 401(k) eligibility; and the LATE and LATE-T suggest positive and significant effects of 401(k) participation for compliers. The similarity in the low-p case is reassuring as it illustrates that there is little impact of variable selection relative to simply including everything in a low-dimensional setting.

We observe somewhat different results in the Quadratic Spline Plus Interactions specification. For both the ATE and the LATE in the Quadratic Spline Plus Interactions case, we see a substantially larger point estimate without selection than with selection, with the selection results being similar to those obtained in the low-p case. Along with the larger point estimate, we also see that the estimated standard errors in the no selection case for the ATE and LATE are roughly three times larger than the standard errors in the selection case. For the ATE-T and LATE-T in the Quadratic Spline Plus Interactions case, point estimates following selection are notably smaller than without selection but estimated standard errors after selection are somewhat larger. We note that one might suspect estimated standard errors for all of the estimators without selection to be substantially downward biased in this case due to the use of many control variables without regularization as in Cattaneo et al. (2010). Finally, we see a large difference in the Orthogonal Polynomials Plus Many Interactions Specifications as estimates cannot even be computed reliably without selection due to severe overfitting: The estimated propensity score is either 0 or 1 for every observation.

We provide estimates of the QTE and QTE-T in Figure 1 and estimates of the LQTE and LQTE-T in Figure 2. The left column of Figure 1 gives results for the QTE, and the right column displays the results for the QTE-T. Similarly, the left and right columns of Figure 2 provide the LQTE and LQTE-T respectively. We give the results for the Quadratic Spline, Quadratic Spline Plus Interactions, and Quadratic Spline Plus Many Interactions specification in the top row, middle row, and bottom row respectively. In each graphic, we use solid lines for point estimates and report uniform 95% confidence intervals with dashed lines.

Looking across the figures, we see a similar pattern to that seen for the estimates of the average effects in that the selection-based estimates are stable across all specifications and are very similar to the estimates obtained without selection from the baseline low-p Quadratic Spline specification. In the more flexible Quadratic Spline plus Interactions specification, the estimates that do not make use of selection behave somewhat erratically. This erratic behavior is especially apparent in the estimated LQTE of 401(k) participation where we observe that small changes in the quantile index may result in large swings in the point estimate of the LQTE and estimated standard errors are quite large. Again, this erratic behavior is likely due to overfitting due to the large set of variables considered. As with the average effects, estimated quantile effects without selection in the Quadratic Spline Plus Many Interactions specification are not reported as the estimated propensity score is always 0 or 1.

If we focus on the LQTE and LQTE-T estimated from variable selection methods, we find that 401(k) participation has a small impact on accumulated net total financial assets at low quantiles while appearing to have a larger impact at high quantiles. Looking at the uniform confidence intervals, we can see that this pattern is statistically significant at the 5% level and that we would reject the hypothesis that 401(k) participation has no effect and reject the hypothesis of a constant treatment effect more generally.

It is also worth discussing the results of the variable selection briefly as well. Due to the number of models and variable selection steps taken, especially in computing quantile effects, it is not practical to give a complete accounting of the selected variables here. Rather, we note that for the linear model, linear IV, ATE, and LATE results, we select between two and 22 variables depending on the specification of controls and left-hand-side variable. The median number of variables selected for the QTE and LQTE results, where the median is taken across index values u, across the different specifications of controls and left-hand-side variables varies between one and 11. There is considerable variability in the number of variables selected across u though, ranging from a minimum of no variables selected to a maximum of 237 selected variables.The selected variables themselves mostly correspond to capturing the effect of income. For example, the union of the variables selected in forming each of the reduced form quantities used for estimating the LATE in the Quadratic Spline Plus Many Interactions specification consists of 36 variables, only four of which do not include income.This pattern of largely selecting terms that are direct income effects or interactions of income with other variables holds up across the specifications considered.

It is interesting that our results are similar to those in Chernozhukov and Hansen (2004) despite allowing for a much richer set of controls. The fact that we allow for a rich set of controls but produce similar results to those previously available lends further credibility to the claim that previous work controlled adequately for the available observables.Finally, it is worth noting that this similarity is not mechanical or otherwise built in to the procedure. For example, applications in Belloni et al. (2012) and Belloni et al. (2014a) use high-dimensional variable selection methods and produce sets of variables that differ substantially from intuitive baselines.

A.1. Overall Notation. We consider a random element taking values in the measure space (), with probability law . Note that it is most convenient to think about P as a parameter in a parameter set P. We shall also work with a bootstrap multiplier variable taking values in () that is independent of , having probability law , which is fixed throughout. We consider (to be i.i.d. copies of , which are also independent of each other. The data will be defined as some measurable function of i = 1, ..., n, where n denotes the sample size.

We require the sequences (to live on a probability space (Ω; note that other variables arising in the proofs do not need to live on the same space. It is important to keep track of the dependence on P in the analysis since we want the results to hold uniformly in P in some set which may be dependent on n. Typically, this set will increase with

Throughout the paper we signify the dependence on P by mostly using P as a subscript in Pbut in the proofs we sometimes use it as a subscript for variables as in . The operator E denotes a generic expectation operator with respect to a generic probability measure P, while Ethe expectation with respect to P. Note also that we use capital letters such as W to denote random elements and use the corresponding lower case letters such as w to denote fixed values that these random elements can take.

We denote by the (random) empirical probability measure that assigns probability each denotes the expectation with respect to the empirical measure, and denotes the empirical process

indexed by a measurable class of functions ; see van der Vaart and Wellner (1996, chap. 2.3). We shall often omit the index and simply write . In what follows, we use to denote the ) norm; for example, we use . For a vector denotes the denotes the Euclidean norm of denotes the which equals the number of non-zero components of v. For a positive integer k, [k] denotes the set denoting sequences in R, the statement means that for some constant A that does not depend on n.

We say that a collection of random variables indexed by a set T and viewed as functions of with respect to W if it is image admissible Suslin class, as defined in Dudley (1999, p 186). In particular, F is suitably measurable if is measurable and T is a Polish space equipped with its Borel sigma algebra, see Dudley (1999, p 186). This condition is a mild assumption satisfied in practical cases.

A.2. Notation for Stochastic Convergence Uniformly in P. All parameters, such as the law of the data, are indexed by P. This dependency is sometimes kept implicit. We shall allow for the possibility that the probability measure can depend on n. We shall conduct our stochastic convergence analysis uniformly in P, where P can vary within some set , which itself may vary with n.

The convergence analysis, namely the stochastic order relations and convergence in distribution, uniformly in and the analysis under all sequences are equivalent. Specifically, consider a sequence of stochastic processes and a random element , taking values in the normed space D, defined on the probability space (Ω). Through most of the Appendix ), the space of uniformly bounded functions mapping an arbitrary index set U to the real line, or D = UC(U), the space of uniformly continuous functions mapping an arbitrary index set U to the real line. Consider also a sequence of deterministic positive constants . We shall say that

Here the symbol denotes weak convergence, i.e. convergence in distribution or law, BLdenotes the space of functions mapping D to [0, 1] with Lipschitz norm at most 1, and the outer probability and expectation, P, are invoked whenever (non)-measurability arises.

Lemma A.1. The above notions (i), (ii) and (iii) are equivalent to the following notions (a), (b), and (c), each holding for every sequence

The claims follow straightforwardly from the definitions, so the proof is omitted. We shall use this equivalence extensively in the proofs of the main results without explicit reference.

B.1. be a sequence of i.i.d. copies of the random element W taking values in the measure space () according to the probability law P on that space. Let be a set of suitably measurable functions W to R, equipped with a measurable envelope . The class is indexed by and is a fixed, totally bounded semi-metric space equipped with a semi-metric Let ) denote the -covering number of the class of functions with respect to the a finitely-discrete measure on (). We shall use the following result.

Theorem B.1 (Uniform in P Donsker Property). Work with the set-up above. Suppose that for q > 2

Furthermore, suppose that

Let denote the P-Brownian Bridge, and consider

(a) Then, uniformly in

(b) The process is stochastically equicontinuous uniformly in , i.e., for every

(c) The limit process has the following continuity properties:

(d) The paths are a.s. uniformly continuous on (under each

Comment B.1. [Important Feature of the Theorem] This is an extension of the uniform Donsker theorem stated in Theorem 2.8.2 in van der Vaart and Wellner (1996), which allows for the function classes F to be dependent on P. This generalization is crucial and is required in all of our problems.

B.2. Uniform in P Validity of Multiplier Bootstrap. Consider the setting of the preceding subsection. Let (be i.i.d multipliers whose distribution does not depend on P, such that E2. Consider the multiplier empirical process:

Here is taken to be an extended empirical processes defined by the empirical measure that assigns mass 1/n to each point (defined in Theorem B.1.

Theorem B.2 (Uniform in P Validity of Multiplier Bootstrap). Assume the conditions of Theorem B.1 hold. Then (a) the following unconditional convergence takes place, uniformly in

and (b) the following conditional convergence takes place, uniformly in , namely uniformly in

where Edenotes the expectation over the multiplier weights (holding the data (

B.3. Uniform in P Functional Delta Method and Bootstrap. We shall use the functional delta method, as formulated in van der Vaart and Wellner (1996, Chap. 3.9). Let normed spaces, with tangentially to if there is a continuous linear map

for all sequences

We now define the following notion of the uniform Hadamard differentiability:

Definition B.1 (Uniform Hadamard Tangential Differentiability). Consider a map , where the domain of the map is a subset of a normed space D and the range is a subset of the normed space be a normed space, with be a compact metric space, a subset of is called Hadamard-differentiable uniformly in tangentially to with derivative map

for all convergent sequences As a part of the definition, we require that the derivative map is linear for each

Comment B.2. Note that the definition requires that the derivative map (ping , is continuous at each (

Comment B.3 (Important Details of the Definition). Definition B.1 is different from the definition of uniform differentiability given in van der Vaart and Wellner (1996, p. 379, eq. (3.9.12)), since our definition allows to be much smaller than and allows to be endowed with a much stronger metric than the metric induced by the norm of D. These differences are essential for infinite-dimensional applications. For example, the quantile/inverse map is uniformly Hadamard differentiable in the sense of Definition B.1 for a suitable choice of be a compact subset of ) such that each is a positive constant. However, the quantile/inverse map is not Hadamard differentiable uniformly on is not uniformly differentiable in the sense of the definition given in van der Vaart and Wellner (1996) which requires . It is important and practical to keep the distinction between and since the estimated values may well be outside unless explicitly imposed in estimation even though the population values of by assumption. For example, the empirical cdf is in , but is outside

be Hadamard-differentiable uniformly in tangentially to with derivative map be a sequence of stochastic processes taking values in , where each estimator of the parameter . Suppose there exists a sequence of constants that uniformly in . The limit process is separable and takes its values in Moreover, the set of stochastic processes is relatively compact in the topology of weak convergence in , that is, every sequence in this set can be split into weakly convergent subsequences. Then, uniformly in is defined and continuous on the whole of , then the sequence converges to zero in outer probability uniformly in . Moreover, the set of stochastic processes is relatively compact in the topology of weak convergence in E.

The following result on the functional delta method applies to any bootstrap or other simulation method obeying certain conditions. Such methods include the multiplier bootstrap as a special case. Let denote the data vector and be a vector of random variables used to generate bootstrap or simulation draws (the specifics may vary depending on the particular method employed). Consider sequences of stochastic processes where in the normed space D uniformly in . Also consider the bootstrap stochastic process is a measurable function of for each value of . Suppose that converges conditionally given in distribution to uniformly in , namely that

uniformly in denotes the expectation computed with respect to the law of holding the data fixed. This is denoted as “uniformly in .” Finally, let denote the bootstrap or simulation draw of

Theorem B.4 (Uniform in P functional delta-method for bootstrap and other simulation methods). Assume the conditions of Theorem B.3 hold. Let indicated previously taking values in in D uniformly in uniformly in

B.4. Proof of Theorem B.1. Part (a) and (b) are a direct consequence of Lemma M.1. In particular, Lemma M.1(a) implies stochastic equicontinuity under arbitrary subsequences which implies part (b). Part (a) follows from Lemma M.1(b) by splitting an arbitrary sequence into subsequences along each of which the covariance function (converges uniformly and therefore also pointwise to a uniformly continuous function on (). This convergence is possible because a relatively compact set in ) in view of the Arzela-Ascoli Theorem, the assumptions in equation (B.1), and total boundedness of (). By Lemma M.1(b) pointwise convergence of the covariance function implies weak convergence to a tight Gaussian process which may depend on the identity of the subsequence. Since this argument applies to each such subsequence that split the overall sequence, part (b) follows.

Part (c) is immediate from the imposed uniform covering entropy condition and Dudley’s metric entropy inequality for expectations of suprema of Gaussian processes (Corollary 2.2.8 in van der Vaart and Wellner (1996)). Claim (d) follows from claim (c) and a standard argument, based on the application of the Borel-Cantelli lemma. Indeed, let be a sequence and 2then by the Markov inequality PThis sums to a finite number over . Hence, by the Borel-Cantelli lemma, for almost all states 2sufficiently large. Hence claim (d) follows.

B.5. Proof of Theorem B.2. Claim (a) is verified by invoking Theorem B.1. We begin by showing that is equal in distribution to , in particular, and share identical mean and covariance function, and thus they share the continuity properties established in Theorem B.1. This claim is immediate from the fact that multiplication by yields a set of measurable functions . Each such function has mean zero under covariance function (. Hence the Gaussian process (the zero mean and the covariance function of (

We are claiming that ) uniformly in We note that the function class and the corresponding envelope satisfy the conditions of Theorem B.1. The same is also true for the function class defined by (maps and its envelope is independent of W. Let Q now denote a finitely discrete measure over . By Lemma K.1 multiplication by does not change qualitatively the uniform covering entropy bound:

Moreover, multiplication by does not affect the norms, is independent of W by construction and E

Claim (b). For each ) denote a closest element in a given, finite over T. We begin by noting that

and the second assertion holds by Theorem B.1 (c).

Second, EThe first assertion follows because Eis bounded by

The second assertion holds by part (a) of the present theorem.

Define Then, by Markov’s inequality, followed by

Finally, by Lemma B.1, for each 0, lim sup

We can now conclude. Note that 0, which holds by the definition of property sup0 noted above. Hence for each 0 and all 0 that 3

Sending 0 gives the result.

B.6. on the following lemma, which is apparently new. An analogous result can be derived for almost sure convergence from well-known non-uniform multiplier central limit theorems, but this strategy requires us to put all the variables indexed by P on the single underlying probability space, which is much less convenient in applications.

Lemma B.1 (Conditional Multiplier Central Limit Theorem in uniformly in (random vectors on , indexed by a parameter The parameter P represents probability laws on , these vectors are assumed to be independent of the i.i.d. sequence (. There exist constants 2 0 , such that Euniformly for all . Then, for every

where Edenotes the expectation over (

Proof of Lemma B.1. Let X and Y be random variables in , then define supIt suffices to show that for any sequence 0 in probability (under P

Following Bickel and Freedman (1981), we shall rely on the Mallow’s metric, written a metric on the space of distribution functions on . For our purposes it suffices to recall that given a sequence of distribution functions and a distribution function 0 if and only iffor each continuous and bounded See Bickel and Freedman (1981) for the definition of

Under the assumptions of the lemma, we can split the sequence into subsequences , along each of which the distribution function of converges to some distribution function with respect to the Mallow’s metric , for some 2 < r < q. This also implies that ) converges weakly to a normal limit ) such that can depend on the subsequence

Let be the empirical distribution function of a sequence (of constant vectors in where . The law of is completely determined by and the law of (the latter is fixed, so it does not enter as the subscript in the definition of as 0 by Lindeberg’s central limit theorem.

Let denote the empirical distribution function of (. By the law of large numbers for arrays, ) in probability along the subsequence in probability along the same subsequence. We can conclude that 0 in prob- ability along the same subsequence by the extended continuous mapping theorem (van der Vaart and Wellner, 1996, Theorem 1.11.1).

The argument applies to every subsequence of the stated form. The claim in the first paragraph of the proof thus follows.

B.7. be a sequence of i.i.d. copies of the random element W taking values in the measure space (law is determined by the probability measure ) be measurable functions indexed by and a fixed, totally bounded semi-metric space (). Consider the stochastic process

This empirical process is indexed by a class of functions with a measurable envelope function . It is important to note here that the dependence on n allows us to have the class itself be possibly dependent on the law

Lemma B.2 (Donsker Theorem for Classes Changing with n). Work with the set-up above. Suppose that for some fixed constant q > 2 and every sequence

(a) Then the empirical process (is asymptotically tight in stochastically equicontinuous. (b) For any subsequence such that the covariance function converges pointwise on converges in to a Gaussian process with covariance function given by the limit of the covariance function along that subsequence.

Proof. The use of Theorem 2.11.1 in van der Vaart and Wellner (1996), which does allow for the probability space to depend on n, allows us to establish claim (a), by repeating the proof (verbatim) of Theorem 2.11.22 in van der Vaart and Wellner (1996, p. 220-221), except that the probability law is allowed to depend on n. (For the sake of completeness, the Supplementary Appendix, provides the complete proof). The proof of claim (b) follows by a standard argument from the stochastic equicontinuity established in claim (a) and finite-dimensional convergence along the indicated subsequences.

B.8. Proof of Theorems B.3 and B.4. The proof consists of two parts, each proving the corresponding theorem.

Part 1. We can split N into subsequences along each of which can possibly depend on . It suffices to verify that for each

where the last two claims hold provided that () is defined and continuous on the whole of . The claim (B.5) is not needed in Part 1, but we need it for Part 2.

The map satisfies ) for every subsequence ). Application of the extended continuous mapping theorem (van der Vaart and Wellner, 1996, Theorem 1.11.1) yields (B.3).

Similarly, the map ). Application of the extended continuous mapping theorem (van der Vaart and Wellner, 1996, Theorem 1.11.1) yields (B.4). The proof of (B.5) is completely analogous and is omitted.

To establish relative compactness, we work with each satisfies ) for every subsequence ). Application of the extended continuous mapping theorem (van der Vaart and Wellner, 1996, Theorem 1.11.1) yields that

Part 2. We can split N into subsequences as above. Along each

where is a separable process in (which is given by plus its independent copy ¯note that ) converge weakly unconditionally to ( ¯) by a standard argument.

Given each the proof is similar to the proof of Theorem 3.9.15 of van der Vaart and Wellner (1996). We can assume without loss of generality that the derivative is defined and continuous on the whole of D. Otherwise, if is defined and continuous only on , we can extend it to D by a Hahn-Banach extension such that ; see van der Vaart and Wellner (1996, p. 380) for details. For each , by claim (B.5), applied to replacing

Subtracting these equations conclude that for each

For every ), the function is contained in BL). Moreover, in D uniformly in along the subsequence . These two facts imply that

Next for each 0 and along

where the (1) conclusion follows by the Markov inequality and by (B.6). Conclude that

Let (be a sequence of i.i.d. copies of random element W taking values in the measure space () according to probability law P. Let F be a set of suitably measurable functions , equipped with a measurable envelope

The following maximal inequality is due to Chernozhukov et al. (2012).

Lemma C.1 (A Maximal Inequality). Work with the setup above. Suppose that is a measurable envelope with any positive constant such that sup. Suppose that there exist constants

where K is an absolute constant. Moreover, for every , with probability

where K(q) > 0 is a constant depending only on q. In particular, setting probability

where is a constant depending only on q and c.

These results follow from the application of results given in Section 5. The details are given in the Supplementary Appendix.

E.1. Proof of Theorem 5.1. In the proof means that , where the constant A depends on the constants in Assumptions 5.1–5.3, but not on , and not on Since the argument is asymptotic, we can assume that in what follows. In order to establish the result uniformly in , it suffices to establish the result under the probability measure induced by any sequence . In the proof we shall use P, suppressing the dependency of on the sample size n. Also, let

Step 1. (A Preliminary Rate Result). In this step we claim that wp 1By definition

for defined in Step 2 below. The bound in (E.2) follows from Step 2 and from the assumption ). Since by Assumption 5.1(iv), 2) does not exceed the

left side of (E.2) and inf) is bounded away from zero uniformly in n, we conclude that sup

Step 2. (Define and bound ) We claim that with probability 1

To establish this, we can bound , where with probability 1

These bounds in turn hold by the following arguments. In order to bound we employ Taylor’s expansion and the triangle inequality. For ¯) denoting a point on a line connecting vectors denoting the mth element of the vector t,

where the last inequality holds by the definition of B(W) given earlier and H¨older’s inequality. By Assumption 5.2(ii)(c), , and by Assumption 5.3, suphence we conclude that

In order to bound we employ the maximal inequality of Lemma C.1 to the class

defined in Assumption 5.3 and equipped with an envelope , to conclude that with probability 1

Here we use that log sup0 by Assumption 5.3; and supby Assumption 5.2(i); Assumption 5.3.

where the terms ) are defined in Step 4 and D) is treated in the next paragraph. Then by the triangle inequality for all and Steps 4 and 5 we have

where the (1) bound follows from Step 4, (1) by assumption, and Step 5.

Moreover, by the orthogonality condition:

Conclude using Assumption 5.1(iv) that

Furthermore, the empirical process (is equivalent to an em- pirical process indexed by -th element of

)) and we make explicit the dependence of ) element of the matrix is a class of uniformly H¨older continuous functions on () with a uniform covering entropy bounded by and equipped with a constant envelope C, given the stated assumptions. This result follows from the fact that by Assumption 5.2(ii)(b)

and the constant envelope follows by Assumption 5.1(iv). Since is generated as a finite sum of products of the elements of M and the class defined in Assumption 5.2, the properties of M and the conditions on in Assumption 5.2(ii) imply that has a uniformly well-behaved uniform covering entropy by Lemma K.1, namely

where is an envelope for Assumption 5.2(i). The class is therefore Donsker uniformly in P because supis bounded by Assumption 5.2(ii), and supby Assumption 5.2(b) and (E.3). Application of Theorem B.1 gives the results of the theorem.

Step 4. (Define and Bound (

and ¯) is a vector on the line connecting

First, by Assumptions 5.2(ii)(d) and 5.3, the claim of Step 1, and the H¨older inequality,

Second, we have that with probability 1 where, for Θ

since supby Assumption 5.3; by Assumption 5.2(i); log sup0 by Lemma K.1 because defined in Assumptions 5.2(i) and 5.3; and can be chosen so that sup

where the first inequality follows by the law of iterated expectations; the second inequality follows by Assumption 5.2(ii)(a); and the last inequality follows from 2] by Assumption 5.2, the monotonicity of the norm ], and Assumption 5.3.

where the first term on the right side is zero by definition of ¯

E.2. Proof of Theorem 5.2. Step 0. In the proof means that , where the constant A depends on the constants in Assumptions 5.1– 5.3, but not on , and not on In Step 1, we consider a sequence , but for simplicity, we write throughout the proof, suppressing the index n. Since the argument is asymptotic, we can assume that what follows.

Let denote the measure that puts mass at the points (denote the expectation with respect to this measure, so that denote the corresponding empirical process

Recall that we define the bootstrap draw as:

where

Step 1.(Linearization) In this step we establish that

The claim would follow from demonstrating that (a)

where )) (not that the hat from disappeared) ; and (b)

To show claim (E.7), we note that with probability 1

where ˜-th element of -th element of )). By the arguments similar to those employed in the proof of the previous

theorem,

for an envelope . By Lemma K.1, multiplication of this class by does not change the entropy bound modulo an absolute constant, namely

Also E[exp(implies (E[max, so that, using independence of (from (and Assumption 5.2(i),

Applying Lemma C.1,

for sup, where the details of calculations are similar to those in the proof of Theorem 5.1. Indeed, with probability 1

where the first inequality follows from the triangle inequality and the law of iterated expectations; the second inequality follows by Assumption 5.2(ii)(a) and Assumption 5.2(i); the third inequality follows from 2] by Assumption 5.2, the monotonicity of the norm Assumption 5.3; and the last inequality follows from by the definition of Θ. The claim (E.7) follows.

To show claim (E.8), bound

since sup(1) by the assumption of the theorem, and since by , which follows by claim (E.7) and by D holding by Theorem B.2.

Step 2. Here we are claiming that , under any sequence . By the triangle inequality and Step 1,

where the first term is by Theorem B.2, and the second term is because (1) implies that E0, which in turn implies that E(1) by the Markov inequality.

E.3. Proof of Theorem 5.3. This is an immediate consequence of Theorems 5.1, 5.2, B.3, and B.4.

In this section, we provide details about how we implemented the methodology developed in the main body of the paper in the empirical example. We first discuss estimation of local average treatment effects (LATE) and then extend this discussion to estimation of local quantile treatment effects (LQTE). Estimation of all other quantities proceeds in a similar fashion and so is not discussed.

F.1. Local Average Treatment Effects. Recall that the LATE of treatment D on outcome Y is defined as

for ) defined in equations (2.1) and (2.3) respectively. It then follows by plugging in the definition of ) that we can express the LATE as

To obtain an estimate of the LATE, we thus need estimates of the low-bias moment function given in equation (3.13), estimates of these key quantities can be constructed from estimates of Eand Eis the binary instrument (401(k) eligibility); D is the binary treatment (401(k) participation); X is the set of raw covariates discussed in the empirical section; and Y is net financial assets. In our application, we have Eunless one is eligible. We use Post-Lasso to estimate Epost--penalized logistic regression to estimate E

To estimate E], we postulate that E(1), where f(X) is one of the pre-specified sets of controls discussed in the empirical section with dimension p. Let denote the indices of observations that have apply the formulation of the Post-Lasso estimator given in Belloni et al. (2012) with outcomes and covariates ))) where Φ(the standard normal distribution function. We calculate penalty loadings according to Algorithm A.1 of Belloni et al. (2012) using Post-Lasso coefficient estimates at each iteration and with a the maximum number of iterations set to 15.(1) denote the resulting Post-Lasso estimates of the coefficients using given above and the final set of penalty loadings. We then estimate E(1) for each i = 1, ..., n. We follow the same procedure to obtain estimates of E(0) for each (0) are the Post-Lasso estimates using only the observations with

Estimation of E] proceed similarly replacing Post-Lasso estimation with post--penalized logistic regression. Specifically, we assume that EΛ(1)) where Λ) is the logistic link function. We then obtain estimates of using the post--penalized estimator defined in equations (3.10) and (3.11) based on the logistic link function and with outcomes and covariates defined as above. We set ))) where Φ() is the standard normal distribution function. We calculate penalty loadings using Algorithm 6.1 of the main text with a maximum of 15 iterations. Let (1) denote the resulting post--penalized estimates of the coefficients using above and the final set of penalty loadings. We estimate Efor each i = 1, ..., n. We follow this procedure to obtain estimates of Efor each are the post--penalized coefficient estimates obtained with as the outcome and as covariates using

Using these baseline quantities, we obtain estimates

We then plug these estimates in to obtain

We report both analytic and bootstrap standard error estimates for the LATE. The analytic standard errors are calculated as

We use wild bootstrap weights for obtaining the multiplier bootstrap estimates of the standard errors with 500 bootstrap replications. Specifically, for each b = 1, ..., 500, we calculate a bootstrap estimate of the LATE as

where 2 is the bootstrap draw for multiplier weight for observa- tion i in bootstrap repetition are random numbers generated as iid draws from two independent standard normal random variables. The bootstrap standard error estimate is then the bootstrap interquartile range rescaled with the normal distribution: [25)], where th quantile of the pth quantile of the N(0, 1).

F.2. Local Quantile Treatment Effects. Calculation and inference for LQTE is more cumbersome than for the LATE. We begin by choosing the set over which we would like to look at the LQTE. In our example, we chose to look at quantiles in the interval [0.1, 0.9].

To calculate the LQTE, we first calculate the local average structural function for outcomes ) for a set of u and then invert to obtain estimates of the LQTE. In our example, we chose to look at 95) are respectively the sample 5percentiles of the outcome of interest Y . Since looking at the continuum of values in this interval is infeasible, we discretize the interval and look at . I.e. we set u equal to each percentile of Y between the 5percentiles for a total of 91 different values of u to be considered. For each value of u, we need an estimate of the local average structural function defined in (2.3) for

As with the LATE, we need estimates of E]. We estimate these quantities as we did for the LATE but change the value of the penalty parameter used to reflect the fact that we are now interested in a large set, in theory a continuum, of model selection problems. Specifically, we assume that E(1)) where Λ) is the logistic link function and f(X) is one of the pre-specified sets of controls discussed in the empirical section with dimension p. We then obtain estimates of (1) by using the post--penalized estimator defined in equations (3.10) and (3.11) based on the logistic link function and with outcomes covariates defined as above. We set where Φ() is the standard normal distribution function. We calculate penalty loadings using Algorithm 6.1 with a maximum of 15 iterations. Let (1) denote the resulting post-estimates of the coefficients using given above and the final set of penalty loadings. We estimate E(1)) for each i = 1, ..., n. We follow this procedure to obtain estimates of Eare the post-coefficient estimates obtained with as the outcome and as covariates and 1)). We also still have Eone cannot participate in a 401(k) unless one is eligible. We then plug-in these estimates to obtain

We also need to obtain estimates of ) for each value of u and for

These estimates will depend on the propensity score, E], estimated above and quantities of the form E]. We again approximate this function with E)) and estimate the coefficients combination of d and z and each u using the post--penalized estimator defined in equations (3.10) and (3.11) based on the logistic link function. We set Φ() is the standard normal distribution function. We calculate penalty loadings using Algorithm 6.1 of the main text with a maximum of 15 iterations. We follow this procedure for each u with as the outcome and covariates as the outcome and covariates as the outcome and covariates defined as above to obtain point estimates (0) respectively. We then estimate Eas Λ(1)) for each i = 1, ..., n and obtain estimates of E1] analogously. As before, we have E1)1(estimates to obtain

Estimates of the local average structural (distribution) functions are formed using the estimators defined in the previous two paragraphs as

To obtain LQTE estimates, we then need to invert these local average structural functions. Since we only have the estimated distribution for each d evaluated on the finite grid of points , we do this inversion by linearly interpolating the value of the distribution function between these points to find the value of the outcome associated with each quantile in the set 9] which we denote as The LQTE at point q is then estimated as

For the LQTE, we only report inference based on the multiplier bootstrap using 500 bootstrap replications. For each b = 1, ..., 500, we generate bootstrap weights as for observation i in bootstrap repetition are random numbers generated as iid draws from two independent standard normal random variables. We then use these weights to

form bootstrap estimates of the local average structural functions

where

From these bootstrap estimates of the average structural distribution functions, we obtain bootstrap LQTE estimates as above through inversion by linearly interpolating the value of the distribution function between the finite set of points at which we have estimated values to find the value of the outcome associated with each quantile in the set 9], denoted (. The bootstrap estimate of the LQTE for bootstrap replication b at point q is then . We form bootstrap standard error estimates for the LQTE at each quantile q as

We also use the bootstrap LQTE estimates to obtain the critical values we use when plotting the uniform confidence bands in our example. We form bootstrap t-statistics for each quantile q as ). We then take and use the 95percentile of the bootstrap distribution of as the critical value in constructing the confidence intervals for our figures following for example Chernozhukov et al. (2013).

Abadie, A. (2002): “Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models,” Journal of the American Statistical Association, 97, 284–292.

——— (2003): “Semiparametric Instrumental Variable Estimation of Treatment Response Models,” Journal of Econometrics, 113, 231–263.

Ai, C. and X. Chen (2003): “Efficient Estimation of Models with Conditional Moment Restrictions Containing Unknown Functions,” Econometrica, 71, 1795–1843.

——— (2012): “The semiparametric efficiency bound for models of sequential moment restrictions containing unknown functions,” Journal of Econometrics, 170, 442–457.

Andrews, D. W. (1994a): “Empirical process methods in econometrics,” Handbook of Econometrics, 4, 2247–2294.

Andrews, D. W. K. (1994b): “Asymptotics for semiparametric econometric models via stochastic equicontinuity,” Econometrica, 62, 43–72.

Angrist, J. D. and J.-S. Pischke (2008): Mostly Harmless Econometrics: An Empiricist’s Companion, Princeton University Press.

Bach, F. (2010): “Self-concordant analysis for logistic regression,” Electronic Journal of Statistics, 4, 384–414.

Belloni, A., D. Chen, V. Chernozhukov, and C. Hansen (2012): “Sparse Models and Methods for Optimal Instruments with an Application to Eminent Domain,” Econometrica, 80, 2369–2429, arxiv, 2010.

-Penalized Quantile Regression for High Dimensional Sparse Models,” Annals of Statistics, 39, 82–130.

——— (2013): “Least Squares After Model Selection in High-dimensional Sparse Models,” Bernoulli, 19, 521–547, arXiv, 2009.

Belloni, A., V. Chernozhukov, I. Fernandez-Val, and C. Hansen (2015): “Supplement to “Program Evaluation with High-Dimensional Data”,” Tech. rep., ArXiv.

Belloni, A., V. Chernozhukov, and C. Hansen (2010): “LASSO Methods for Gaussian Instrumental Variables Models,” 2010 arXiv:[math.ST], http://arxiv.org/abs/1012.1297.

——— (2013a): “Inference for High-Dimensional Sparse Econometric Models,” Advances in Economics and Econometrics. 10th World Congress of Econometric Society. August 2010, III, 245– 295.

——— (2014a): “Inference on Treatment Effects After Selection Amongst High-Dimensional Con- trols,” Review of Economic Studies, 81, 608–650.

Belloni, A., V. Chernozhukov, and K. Kato (2013b): “Uniform Post Selection Inference for LAD Regression Models,” arXiv preprint arXiv:1304.0282.

Belloni, A., V. Chernozhukov, and L. Wang (2011): “Square-Root-LASSO: Pivotal Recovery of Sparse Signals via Conic Programming,” Biometrika, 98, 791–806, arxiv, 2010.

Belloni, A., V. Chernozhukov, L. Wang, et al. (2014b): “Pivotal estimation via square-root lasso in nonparametric regression,” The Annals of Statistics, 42, 757–788.

Belloni, A., V. Chernozhukov, and Y. Wei (2013c): “Honest Confidence Regions for Logistic Regression with a Large Number of Controls,” arXiv preprint arXiv:1304.3969.

Benjamin, D. J. (2003): “Does 401(k) eligibility increase saving? Evidence from propensity score subclassification,” Journal of Public Economics, 87, 1259–1290.

Berry, S., J. Levinsohn, and A. Pakes (1995): “Automobile Prices in Market Equilibrium,” Econometrica, 63, 841–890.

Bickel, P. J. (1982): “On adaptive estimation,” The Annals of Statistics, 647–671.

Bickel, P. J. and D. A. Freedman (1981): “Some asymptotic theory for the bootstrap,” The Annals of Statistics, 1196–1217.

Bickel, P. J., Y. Ritov, and A. B. Tsybakov (2009): “Simultaneous analysis of Lasso and Dantzig selector,” Annals of Statistics, 37, 1705–1732.

Cand`es, E. and T. Tao (2007): “The Dantzig selector: statistical estimation when p is much larger than , 35, 2313–2351.

Caner, M. and H. H. Zhang (2014): “Adaptive elastic net for generalized methods of moments,” Journal of Business and Economic Statistics, 32, 30–47.

Cattaneo, M., M. Jansson, and W. Newey (2010): “Alternative Asymptotics and the Partially Linear Model with Many Regressors,” Working Paper, http://econ-www.mit.edu/files/6204.

Cattaneo, M. D. (2010): “Efficient semiparametric estimation of multi-valued treatment effects under ignorability,” Journal of Econometrics, 155, 138–154.

Chamberlain, G. (1992): “Efficiency Bounds for Semiparametric Regression,” Econometrica, 60, 567–596.

Chamberlain, G. and G. W. Imbens (2003): “Nonparametric applications of Bayesian inference,” Journal of Business & Economic Statistics, 21, 12–18.

Chen, X. (2007): “Large Sample Sieve Estimatin of Semi-Nonparametric Models,” Handbook of Econometrics, 6, 5559–5632.

Chen, X., O. Linton, and I. v. Keilegom (2003): “Estimation of Semiparametric Models when the Criterion Function Is Not Smooth,” Econometrica, 71, 1591–1608.

Chernozhukov, V., D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, and a. W. Newey (2016): “Double Machine Learning for Treatment and Causal Parameters,” ArXiv e-prints.

Chernozhukov, V., D. Chetverikov, and K. Kato (2012): “Gaussian approximation of suprema of empirical processes,” ArXiv e-prints.

(2005): “Subsampling inference on quantile regression processes,” , 67, 253–276.

(2013): “Inference on counterfactual distributions,” Econometrica, 81, 2205–2268.

Chernozhukov, V. and C. Hansen (2004): “The impact of 401(k) participation on the wealth distribution: An instrumental quantile regression analysis,” Review of Economics and Statistics, 86, 735–751.

——— (2005): “An IV Model of Quantile Treatment Effects,” Econometrica, 73, 245–262.

——— (2006): “Instrumental quantile regression inference for structural and treatment effect mod- els,” J. Econometrics, 132, 491–525.

Chernozhukov, V., C. Hansen, and M. Spindler (2015a): “Post-Selection and PostRegularization Inference in Linear Models with Very Many Controls and Instruments,” American Economic Review: Papers and Proceedings, 105, 486–490.

——— (2015b): “Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach,” Annual Review of Economics, 7, 649–688.

Chesher, A. (2003): “Identification in nonseparable models,” Econometrica, 71, 1405–1441.

Dudley, R. M. (1999): Uniform central limit theorems, vol. 63 of Cambridge Studies in Advanced Mathematics, Cambridge: Cambridge University Press.

Engen, E. M. and W. G. Gale (2000): “The Effects of 401(k) Plans on Household Wealth: Differences Across Earnings Groups,” Working Paper 8032, National Bureau of Economic Research.

Engen, E. M., W. G. Gale, and J. K. Scholz (1996): “The Illusory Effects of Saving Incentives on Saving,” Journal of Economic Perspectives, 10, 113–138.

Escanciano, J. C. and L. Zhu (2013): “Set inferences and sensitivity analysis in semiparametric conditionally identified models,” CeMMAP working papers CWP55/13, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.

Fan, J. and R. Li (2001): “Variable selection via nonconcave penalized likelihood and its oracle properties,” Journal of American Statistical Association, 96, 1348–1360.

Farrell, M. (2015): “Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations,” Journal of Econometrics, 174, 1–23.

Frank, I. E. and J. H. Friedman (1993): “A Statistical View of Some Chemometrics Regression Tools,” Technometrics, 35, 109–135.

(2013): “Identification of treatment effects on the treated with one-sided non-compliance,” Econometric Reviews, 32, 384–414.

Ghosal, S., A. Sen, and A. W. van der Vaart (2000): “Testing Monotonicity of Regression,” Ann. Statist., 28, 1054–1082.

(1984): “Some limit theorems for empirical processes,” Ann. Probab., 12, 929–998, with discussion.

Hahn, J. (1997): “Bayesian bootstrap of the quantile regression estimator: a large sample study,” Internat. Econom. Rev., 38, 795–808.

——— (1998): “On the role of the propensity score in efficient semiparametric estimation of average treatment effects,” Econometrica, 315–331.

Hansen, B. E. (1996): “Inference when a nuisance parameter is not identified under the null hypothesis,” Econometrica, 64, 413–430.

Hansen, L. P. (1982): “Large sample properties of generalized method of moments estimators,” Econometrica, 50, 1029–1054.

Hansen, L. P. and K. J. Singleton (1982): “Generalized instrumental variables estimation of nonlinear rational expectations models,” Econometrica, 50, 1269–1286.

Heckman, J. and E. J. Vytlacil (1999): “Local instrumental variables and latent variable models for identifying and bounding treatment effects,” Proc. Natl. Acad. Sci. USA, 96, 4730– 4734 (electronic).

Heckman, J. J. and E. Vytlacil (2005): “Structural equations, treatment effects, and econometric policy evaluation,” Econometrica, 73, 669–738.

Hong, H. and D. Nekipelov (2010): “Semiparametric efficiency in nonlinear LATE models,” Quantitative Economics, 1, 279–304.

Hong, H. and O. Scaillet (2006): “A fast subsampling method for nonlinear dynamic models,” J. Econometrics, 133, 557–578.

Huang, J., J. L. Horowitz, and S. Ma (2008): “Asymptotic properties of bridge estimators in sparse high-dimensional regression models,” The Annals of Statistics, 36, 587613.

Huang, J., J. L. Horowitz, and F. Wei (2010): “Variable selection in nonparametric additive models,” Ann. Statist., 38, 2282–2313.

Imbens, G. W. and J. D. Angrist (1994): “Identification and Estimation of Local Average Treatment Effects,” Econometrica, 62, 467–475.

Imbens, G. W. and W. K. Newey (2009): “Identification and estimation of triangular simultaneous equations models without additivity,” Econometrica, 77, 1481–1512.

Imbens, G. W. and D. B. Rubin (2015): Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, Cambridge University Press.

Jing, B.-Y., Q.-M. Shao, and Q. Wang (2003): “Self-normalized Cramr-type large deviations for independent random variables,” Ann. Probab., 31, 2167–2215.

Kato, K. (2011): “Group Lasso for high dimensional sparse quantile regression models,” Preprint, ArXiv.

Kline, P. and A. Santos (2012): “A Score Based Approach to Wild Bootstrap Inference,” Journal of Econometric Methods, 1, 23–41.

Koenker, R. (1988): “Asymptotic Theory and Econometric Practice,” Journal of Aplpied Econometrics, 3, 139–147.

——— (2005): Quantile regression, Cambridge university press.

Kosorok, M. R. (2008): Introduction to Empirical Processes and Semiparametric Inference, Series in Statistics, Berlin: Springer.

(2008a): “Can one estimate the unconditional distribution of post-model-selection estimators?” Econometric Theory, 24, 338–376.

——— (2008b): “Recent developments in model selection and related areas,” Econometric Theory, 24, 319–322.

Linton, O. (1996): “Edgeworth approximation for MINPIN estimators in semiparametric regression models,” Econometric Theory, 12, 30–60.

Mammen, E. (1993): “Bootstrap and wild bootstrap for high dimensional linear models,” The Annals of Statistics, 255–285.

Meinshausen, N. and B. Yu (2009): “Lasso-type recovery of sparse representations for high-dimensional data,” Annals of Statistics, 37, 2246–2270.

Newey, W. K. (1990): “Semiparametric efficiency bounds,” Journal of Applied Econometrics, 5, 99–135.

——— (1994): “The asymptotic variance of semiparametric estimators,” Econometrica, 62, 1349– 1382.

——— (1997): “Convergence Rates and Asymptotic Normality for Series Estimators,” Journal of Econometrics, 79, 147–168.

) tests and their use,” Sankhya, 41, 1–21.

Ogburn, E. L., A. Rotnitzky, and J. M. Robins (2015): “Doubly robust estimation of the local average treatment effect curve,” Journal of the Royal Statistical Society: Series B, 77, 373–396.

Poterba, J. M., S. F. Venti, and D. A. Wise (1994): “401(k) Plans and Tax-Deferred savings,” in Studies in the Economics of Aging, ed. by D. A. Wise, Chicago, IL: University of Chicago Press.

——— (1995): “Do 401(k) Contributions Crowd Out Other Personal Saving?” Journal of Public Economics, 58, 1–32.

——— (1996): “Personal Retirement Saving Programs and Asset Accumulation: Reconciling the Evidence,” Working Paper 5599, National Bureau of Economic Research.

——— (2001): “The Transition to Personal Accounts and Increasing Retirement Wealth: Macro and Micro Evidence,” Working Paper 8610, National Bureau of Economic Research.

(2009): “Confidence Sets Based on Sparse Estimators Are Necessarily Large,” Sankhya, 71-A, 1–18.

Robins, J. M. and A. Rotnitzky (1995): “Semiparametric efficiency in multivariate regression models with missing data,” J. Amer. Statist. Assoc., 90, 122–129.

Robinson, P. M. (1988): “Root-N-consistent semiparametric regression,” Econometrica, 56, 931– 954.

Romano, J. P. and A. M. Shaikh (2012): “On the uniform asymptotic validity of subsampling and the bootstrap,” The Annals of Statistics, 40, 2798–2822.

Rothe, C. and S. Firpo (2013): “Semiparametric Estimation and Inference Using Doubly Robust Moment Conditions,” Tech. rep., NYU preprint.

Sherman, R. (1994): “Maximal inequalities for degenerate U-processes with applications to optimization estimators,” Ann. Statist., 22, 439–459.

Spindler, M., V. Chernozhukov, and C. Hansen (2016): hdm: High-Dimensional Metrics, R package version 0.1.0, http://CRAN.R-project.org/package=hdm.

Tibshirani, R. (1996): “Regression shrinkage and selection via the Lasso,” J. Roy. Statist. Soc. Ser. B, 58, 267–288.

Tsybakov, A. B. (2009): Introduction to nonparametric estimation, Springer.

van de Geer, S. A. (2008): “High-dimensional generalized linear models and the lasso,” Annals of Statistics, 36, 614–645.

van der Vaart, A. W. (1991): “On differentiable functionals,” The Annals of Statistics, 178–204.

——— (1998): Asymptotic Statistics, Cambridge University Press.

van der Vaart, A. W. and J. A. Wellner (1996): Weak Convergence and Empirical Processes, Springer Series in Statistics.

Vytlacil, E. J. (2002): “Independence, Monotonicity, and Latent Index Models: An Equivalence Result,” Econometrica, 70, 331–341.

Wasserman, L. (2006): All of nonparametric statistics, Springer New York.

Wooldridge, J. M. (2010): Econometric Analysis of Cross Section and Panel Data, Cambridge, Massachusetts: The MIT Press, second ed.

Zou, H. (2006): “The Adaptive Lasso And Its Oracle Properties,” Journal of the American Statistical Association, 101, 1418–1429.

Supplement to “Program Evaluation and Causal Inference with High-Dimensional Data”

Abstract. The supplementary material contains 10 appendices with additional results and some omitted proofs. Appendices F–J include additional results for Sections 2–7, respectively. Appendix K gathers auxiliary results on algebra of covering entropies. Appendices L and M contain the proofs of Sections 4 and 5 omitted from the main text. Appendix N contains the proofs of Sections 6 omitted from the main text, together with the proofs of the additional results for Section 6 in Appendix I. Appendix O reports the results of a simulation experiment.

F.1. Causal Interpretations for Structural Parameters. The quantities discussed in Sections 2.2 and 2.3 are well-defined and have causal interpretation under standard conditions. We briefly recall these conditions, using the potential outcomes notation. Let denote the potential outcomes under the treatment states 1 and 0. These outcomes are not observed jointly, and we instead observe is the random variable indicating program participation or treatment state. Under exogeneity, D is assigned independently of the potential outcomes conditional on covariates a.s., where statistical independence.

Exogeneity fails when D depends on the potential outcomes. For example, people may drop out of a program if they think the program will not benefit them. In this case, instrumental variables are useful in creating quasi-experimental fluctuations in D that may identify useful effects. Let Z be a binary instrument, such as an offer of participation, that generates potential participation decisions under the instrument states 1 and 0, respectively. As with the potential outcomes, the potential participation decisions under both instrument states are not observed jointly. The realized participation decision is then given by We assume that Z is assigned randomly with respect to potential outcomes and participation decisions conditional on

There are many causal quantities of interest for program evaluation. Chief among these are various structural averages: ], the causal ASF; ], the causal LASF; and the causal LASF-T; as well as effects derived from them such as E], the causal ATE; Ethe causal ATE-T; E], the causal LATE; and Ethe causal LATE-T. These causal quantities are the same as the structural parameters defined in Sections 2.2-2.3 under the following well-known sufficient condition.

Assumption F.1 (Assumptions for Causal/Structural Interpretability). The following conditions hold P-almost surely: (Exogeneity) ((; (First Stage) EE; (Non-Degeneracy) P; (Monotonicity) P

This condition due to Imbens and Angrist (1994) and Abadie (2003) is much-used in the program evaluation literature. It has an equivalent formulation in terms of a simultaneous equation model with a binary endogenous variable; see Vytlacil (2002) and Heckman and Vytlacil (1999). For a thorough discussion of this assumption, we refer to Imbens and Angrist (1994). Using this assumption, we present an identification lemma which follows from results of Abadie (2003) and Hong and Nekipelov (2010) that both in turn build upon Imbens and Angrist (1994). The lemma shows that the parameters defined earlier have a causal interpretation under Assumption F.1. Therefore, our referring to them as structural/causal is justified under this condition.

Lemma F.1 (Identification of Causal Effects )

Furthermore, if D is exogenous, namely a.s., then

Comment G.1 (Another strategy for estimating An alternative to the strategy for modeling and estimating is to treat as in the text via (3.7) while modeling through its disaggregation

where the regression functions map the support of (D, Z, X), DZX, to the real line and are defined by

We will denote other potential values for the functions by the parameters e and l. In this alternative approach, we can again use high-dimensional methods for modeling and estimating using the same approach as in the main paper, and we can then use the relation (G.1) to estimate Specifically, we model the conditional expectation of V given D, Z, and X by

We model the conditional probability of D taking on 1 or 0, given Z and X by

As in the strategy in the main text, we maintain approximate sparsity. We assume that there exist such that, for all

That is, there are at most components of with nonzero values in the approximations to . The sparsity condition also requires the size of the approximation errors to be small compared to the conjectured size of the estimation error: For all , we assume

Note that the size of the approximating model can grow with n just as in standard series estimation as long as

We proceed with the estimation of analogously to the approach outlined in the main text. The Lasso estimator and Post-Lasso estimator ˜are defined analogously to ˜using the data ( ˜. The estimator , has the near oracle rate of convergence and other desirable properties. The Lasso estimator and Post-Lasso estimators ˜are also defined analogously to using the data ( ˜. Again, the estimator ¯, has good theoretical properties including the near oracle rate of convergence, . The resulting estimator for

The remaining estimation steps are the same as with the strategy given in the main text.

Assumption H.1 (Approximate Sparsity for the Strategy of Section G.1). Under each and for each , uniformly for all : (i) The approximations (G.4)-(G.10) and (3.7) apply with the link functions Γbelonging to the set L, the sparsity condition holding, the approximation errors satisfying , and the sparsity index s and the number of terms p in the vector . (ii) There are estimators ¯that, with probability no less than 1, the estimation errors satisfy ; the estimators are sparse such that empirical and population norms induced by the Gram matrix formed by (are equivalent on sparse subsets, sup. (iii) The following boundedness conditions hold:

Under the stated assumptions, the empirical reduced form process ) defined by (3.16), but constructed using the alternative strategy for estimating of Comment G.1, follows a functional central limit theorem and a functional central limit theorem for the multiplier bootstrap. Theorem H.1 states these results. We omit the proof because it is analogous to the proofs of Theorems 4.1–4.2.

Theorem H.1. Under Assumption H.1 the results stated in Theorems 4.1–4.2 in the main text apply to the alternative strategy for estimating of Comment G.1.

I.1. Assumptions. We consider the following high level conditions which are implied by the primitive Assumptions 6.1 and 6.2. For each 1, there is a sequence of independent random variables (, defined on the probability space (Ω) such that model (6.1) holds with Let be a metric on U (and note that the results cover the case where is a function of n). Throughout this section we assume that the variables (generated as suitably measurable transformations of . Furthermore, this section uses the notation ¯E], because we allow for independent non-identically distributed (i.n.i.d.) data.

Consider fixed sequences of positive numbers 0 at a speed at most polynomial in ; and positive constants c and C which will not vary with P.

; (ii) uniformly over , we have that max

The following technical lemma justifies the choice of penalty level . It is based on self-normalized moderate deviation theory. In what follows, for denote a diagonal of “ideal loadings” with diagonal elements given by

Lemma I.1 (Choice of Suppose Condition WL holds, let be constants, [1large enough depending only on Condition WL,

We note that Condition WL(iii) contains high level conditions on the process (following lemma provides easy to verify sufficient conditions that imply Condition WL(iii).

Lemma I.2. Suppose the i.i.d. sequence ((, satisfies the following conditions: (i) -a.s.; (ii) for some random variable is a VC-class of functions with VC-index equal to , (iii) For some fixed we have Ehave with probability 1

Lemma I.2 allows for several different cases including cases where is generated by a non-smooth transformation of a random variable Y . For example, if has bounded conditional probability density function, we have similar result holds for independent non-identically distributed data.

In what follows for a vector , and a set of indices , we denote by the vector such that (. For a set T, |T| denotes the cardinality of T. Moreover, let

I.2. Finite Sample Results: Linear Case. For the model described in (6.1) with Λ(t) = t and we will study the finite sample properties of the associated Lasso and Post-Lasso estimators of (defined in relations (6.2) and (6.3).

The analysis relies on 1, and on the restricted eigenvalues

and maximum and minimum sparse eigenvalues

Next we present technical results on the performance of the estimators generated by Lasso that are used in the proof of Theorem 6.1.

Lemma I.3 (Rates of Convergence for Lasso)

The following lemma summarizes sparsity properties of (

Lemma I.5 (Rate of Convergence of Post-Lasso). Under Conditions WL, let be the Post-Lasso estimator based on the support . Then, with probability 1 , uniformly over for ˜

Moreover, if supp(, the following events , imply that

I.3. Finite Sample Results: Logistic Case. For the model described in (6.1) with Λ(t) = exp(we will study finite the sample properties of the associated Lasso and Post-Lasso estimators of (in relations (6.2) and (6.3). In what follows we use the notation

In the finite sample analysis we will consider not only the design matrix a weighted counterpart , is the conditional variance of the outcome variable

For a subset , let the non-linear impact coefficient Belloni and Chernozhukov (2011); Belloni et al. (2013c) be defined as

Note that ¯can be bounded as

which can lead to interesting bounds provided is appropriate (like the restrictive set ∆in the definition of restricted eigenvalues). In Lemma I.6 we have . For this choice of sets, and provided that with probability 1 (1) we have sup, we have that uniformly over , with probability

The definitions above differ from their counterpart in the analysis of -penalized least squares estimators by the weighting 0 1. Thus it is relevant to understand their relations through the quantities

Many primitive conditions on the data generating process will imply ) to be bounded away from zero for the relevant choices of A. We refer to Belloni et al. (2013c) for bounds on notational convenience we will also work with a rescaling of the approximation errors ˜as

which is the unique solution to Λ(It follows that

Next we derive finite sample bounds provided some crucial events occur.

Lemma I.6 (Rates of Convergence for -Logistic Estimator). Assume that

for c > 1. Further, let , uniformly over ˜

Provided that the nonlinear impact coefficient ¯

for every , we have uniformly over

The following result provides bounds on the number of non-zero coefficients in the estimator , uniformly over

Lemma I.7 (Sparsity of -Logistic Estimator)

for every , uniformly over

Next we turn to finite sample bounds for the logistic regression estimator where the support was selected based on -penalized logistic regression. The results will hold uniformly over provided the side conditions also hold uniformly over U.

Lemma I.8 (Rate of Convergence for Post--Logistic Estimator)defined as the post model selection logistic regression with the support . Uniformly over have

Comment I.1. Since for a sparse vector

), the results above can directly establish bounds on the rate of convergence in the

In this section, we report additional results to supplement those provided in the main text. Specifically, we provide results with both total wealth and net total financial assets as the outcome variable. We present detailed results for four different sets of controls f(X). The first set uses the indicators of marital status, two-earner status, defined benefit pension status, IRA participation status, and home ownership status, a linear term for family size, five categories for age, four categories for education, and seven categories for income (Indicator specification). We use the same definitions of categories as in Chernozhukov and Hansen (2004) and note that this is identical to the specification in Chernozhukov and Hansen (2004) and Benjamin (2003). The second through fourth specifications correspond to the Quadratic Spline specification, the Quadratic Spline Plus Interactions specification, and the Quadratic Spline Plus Many Interactions specification described in the main text.

Results for intention to treat effects based on using 401(k) eligibility as the treatment variable are given in Appendix Table 1. In Appendix Table 2, we report results using 401(k) participation as the treatment variable instrumenting with 401(k) eligibility. We plot the QTE and QTE-T, based on using 401(k) eligibility as the treatment variable, in Figures 3-6. Finally, the LQTE and LQTE-T, based on using 401(k) participation as the treatment variability and instrumenting with eligibility, are plotted in Appendix Figures 7-10. The results are broadly consistent with the discussion provided in the main text with the selection and no selection results being similar in the low-dimensional cases and the selection results being substantially more regular in the high-dimensional cases. We also see that the patterns of point estimates for total wealth and net total financial assets are similar, though the total wealth estimates have substantially larger estimated standard errors, especially for high quantiles.

Lemma K.1 (Algebra for Covering Entropies). Work with the setup described in Appendix C of the main text.

(1) Let F be a VC subgraph class with a finite VC index k or any other class whose entropy is bounded above by that of such a VC subgraph class, then the covering entropy of F obeys:

(2) For any measurable classes of functions

(3) Given a measurable class F mapping W to R and a random variable taking values in R,

(4) Given measurable classes and envelopes , a function such that for , and fixed functions ¯, the class of functions

Proof. For the proof (1)-(2) see, e.g., Andrews (1994a) and (3) follows from (2). To show (4) let . Then, by the condition on

Let with the measure ). Then the set with respect to the measure Q by (K.1). Thus, for any 0 we have that

Therefore,

and the result follows since the right hand side no longer depends on

Lemma K.2 (Covering Entropy for Classes obtained as Conditional Expectations). Let F denote a class of measurable functions with a measurable envelope F. For a given be the function ¯is a regular conditional probability distribution over conditional on ¯be an envelope for ¯F. Then, for

where Q belongs to the set of finitely-discrete probability measures over belongs to the set of finitely-discrete probability measures over . In particular, for every

Proof. The proof generalizes the proof of Lemma A.2 in Ghosal et al. (2000). For the corresponding ¯, and any probability measure Q on W, by Jensen’s inequality, for any

where ). Therefore, for any

where we use Problems 2.5.1-2 of van der Vaart and Wellner (1996) to replace the supremum over ¯Q with the supremum over finitely-discrete probability measures

Moreover, taking k = 1,

where we use Problems 2.5.1-2 of van der Vaart and Wellner (1996) to replace the supremum over ¯Q with the supremum over finitely-discrete probability measures , and then Problem 2.10.4 of van der Vaart and Wellner (1996) to argue that the last bound in weakly increasing in

Comment K.1. Lemma K.2 extends the result in Lemma A.2 in Ghosal et al. (2000) and Lemma 5 in Sherman (1994) which considered integral classes with respect to a fixed measure our applications we need to allow the integration measure to vary with w, namely we allow for to be a conditional distribution.

L.1. Proof of Theorem 4.1. Step 0. (Preparation). In the proof means that where the constant A depends on the constants in Assumptions 4.1 and 4.2 only, but not on n once , and not on . We consider a sequence simplicity, we write throughout the proof, suppressing the index n. Since the argument is asymptotic, we can assume that in what follows.

To proceed with the presentation of the proofs, it might be convenient for the reader to have the notation collected in one place. The influence function and low-bias moment functions for for are given respectively by

The influence function and the moment function for Recall that the estimator of the reduced-form parameters are solutions to the equations

where ¯are estimators as in Assumption 4.2. For each variable

we obtain the estimator estimator and the estimand are vectors in with a fixed finite dimension. We stack these vectors into the processes

where . The components (of ) trivially have the linear representation (with no error) for each . We only need to establish the claim for the empirical process (, which we do in the steps below.

(a) We make some preliminary observations. For (, we define the function (

The derivatives of this function with respect to t obey for all

where L depends only on

(b) Let

We observe that with probability no less than 1

where

To see this, note that under Assumption 4.2 for all

for , with evaluation after computing the norms, and for denoting suphere and below. Similarly, under Assumption 4.2,

for , with evaluation after computing the norms, and noting that for any

Hence with probability at least 1

(c) We have that

so that

with h evaluated at

(d) Note that for

with h evaluated at after computing the expectations under P.

By the law of iterated expectations and the orthogonality property of the moment condition for

Moreover, uniformly for any , in view of properties noted in Steps (a) and (b),

Since with probability 1

(e) Furthermore, with probability 1

The classes of functions,

viewed as maps from the sample space W to the real line, are bounded by a constant envelope and obey log sup0, which holds by Assumption 4.1(ii), and log sup0 which holds by Assumption 4.1(ii) and Lemma K.2. The uniform covering entropy of the function sets

are trivially bounded by log(e

The class of functions

has a constant envelope and is a subset of

which is a union of 5 sets of the form

with Λ a fixed monotone function for each of the 5 sets; each of these sets are the unions of at mostVC-subgraph classes of functions with VC indices bounded by . Note that a fixed monotone transformations Λ preserves the VC-subgraph property (van der Vaart and Wellner, 1996, Lemma 2.6.18). Therefore

Similarly, the class of functions (1))) has a constant envelope, is a union of at most 5 sets, which are themselves the unions of at mostVC-subgraph classes of functions with VC indices bounded by since a fixed monotone transformations Λ preserves the VC-subgraph property. Therefore, log sup

Finally, the set of functions

is a Lipschitz transform of function sets , with bounded Lipschitz coeffi-cients and with a constant envelope. Therefore,

Applying Lemma C.1 with and the envelope , with probability 1 for some constant K > e

Here we have used some simple calculations, exploiting the boundedness condition in Assumptions 4.1 and 4.2, to deduce that

by definition of the set , so that we can use Lemma C.1. We also note that log(1by the assumption on by Assumption 4.2(i).

(f) The claim of Step 1 follows by collecting Steps (a)-(e).

Step 2 (Uniform Donskerness). Here we claim that Assumption 4.1 implies that the set of vectors of functions (-Donsker uniformly in P, namely that

where . Moreover, has bounded, uniformly continuous paths uniformly in

To verify these claims we shall invoke Theorem B.1.

To demonstrate the claim, it will suffice to consider the set of ]). Further, we notice that

defined in Step 1(e), where the validity of the Lipschitz property relies on Assumption 4.1(iii) (to keep the denominator away from zero) and on the boundedness conditions in Assumption 4.1(iii) and Assumption 4.2(iii). The function sets are uniformly bounded classes that have uniform covering entropy bounded by log(e0 up to a multiplicative constant, and so , which is uniformly bounded under Assumption 4.1, the uniform covering entropy bounded by log(e0 up to a multiplicative constant (e.g. van der Vaart and Wellner (1996)). Since uniformly bounded and is a finite union of function sets with the uniform entropies obeying the said properties, it also follows that has this property; namely,

Sinceis uniformly bounded, the first condition in (B.1) and the entropy condition (B.2) in Theorem B.1 hold.

We demonstrate the second condition in (B.1). Consider a sequence of positive constants approaching zero, and note that

where must be of the form:

with () equal to either (

which we deduce by the definition of ] and the contraction property of the conditional expectation.

L.2. Proof of Theorem 4.2. The proof will be similar to the proof of Theorem 4.1.

Step 0. (Preparation). In the proof means that , where the constant A depends on the constants in Assumptions 4.1 and 4.2 only, but not on not on . We consider a sequence , but for simplicity, we write throughout the proof, suppressing the index n. Since the argument is asymptotic, we can assume that in what follows. Let denote the measure that puts mass on points (Let denote the expectation with respect to this measure, so that denote the corresponding empirical process

Recall that we define the bootstrap draw as:

since is independent of W and has zero mean. Here is a plug-in estimator of the influence function

Step 1.(Linearization) In this step we establish that

where ) is a linearization error, arising completely due to estimation of the influence function; if the influence function were known, this term would be zero.

For the components () the linearization follows by the represen- tation,

for all , and noting that supin (L.3) by Theorem 4.1 and by

It remains to establish the claim for the empirical process (. As in the proof of Theorem 4.1, we have that with probability at least 1

where sup) by Theorem 4.1.

where

By the calculations in Step 1(e) of the proof of Theorem 4.1, obeys log sup(By Lemma K.1, multiplication of this class by does not change the entropy bound modulo an absolute constant, namely

where the envelope times a constant. Also, E[exp(implies that (E[maxThus, applying Lemma C.1 with and the envelope , for some constant K > e

for sup; where the details of calculations are the same as in Step 1(e) of the proof of Theorem 4.1.

Finally, we conclude that

Step 2. Here we are claiming that , under any sequence . We have that

where the first term is by Theorem B.2, and the second term is because (1) implies that E0, which in turn implies that E(1) by the Markov inequality.

L.3. Proof of Corollary 4.1. This is an immediate consequence of Theorems 4.1, 4.2, B.3, and B.4.

Lemma M.1 (Donsker Theorem for Classes Changing with n). Work with the set-up described in Appendix B of the main text. Suppose that for some fixed constant q > 2 and every sequence

(a) Then the empirical process (is asymptotically tight in . (b) For any subsequence such that the covariance function converges pointwise on (converges in to a Gaussian process with covariance function given by the limit of the covariance function along that subsequence.

Proof. The proof follows is similar to the proof of Theorem 2.11.22 in van der Vaart and Wellner (1996, p. 220-221), except that the probability law is allowed to depend on n. Indeed, the use of Theorem 2.11.1 in van der Vaart and Wellner (1996), which does allow for the probability space to depend on n, allows us to establish claim (a), whereas the proof of claim (b) follows by a standard argument.

The random distance given in Theorem 2.11.1 in van der Vaart and Wellner (1996) (Lemma M.2 below) reduces to It follows that )), for every is replaced by 1 , then the conditions of the lemma still hold. Hence, assume without loss of generality than 1. Insert the bound on the covering numbers and next make a change of variables to bound the entropy integralin Lemma M.2 byconverges to zero in probability for every 0 by the conditions of the lemma. Apply Lemma M.2 to obtain the result.

Lemma M.2 (van der Vaart and Wellner (1996, Th. 2.11.1))be independent stochastic processes, defined on the product probability space with each depending on the ith coordinate of , and indexed by a totally bounded semimetric space (. Assume that the sums are measurable in the sense that every one of the maps

is measurable, for every , every vector (, and every natural number n. Also, for every

andis the random semimetric

Then the sequence is asymptotically -equicontinuous.

Proof of Theorem 6.1. In order to establish the result uniformly in , it suffices to establish the result under the probability measure induced by any sequence . In the proof we shall use P, suppressing the dependency of on the sample size n. To prove this result we invoke Lemmas I.3-I.5 in Appendix I. These lemmas rely on specific events (described below) and Condition WL which is also stated in Appendix I. We will show that Assumption 6.1 implies that the required events occur with probability 1 (1) and also implies Condition WL.

Let denote the ideal penalty loadings. The three events required to occur with probability 1 (1) are the following: , and where for some 1uniformly bounded for the penalty loading in all iterations sufficiently large.

By Assumption 6.1(iv)(b) holds with probability 1

Next we verify that Condition WL holds. Condition WL(i) is implied by the approximate sparsity condition in Assumption 6.1(i) and the covering condition in Assumption 6.1(ii). By Assumption 6.1 we have that is fixed and the Algorithm sets and Φby Assumption 6.1(i). Since it is assumed that Euniformly in , Condition WL(ii) holds. Condition WL(iii) follows from Assumption 6.1(iv).

Since Condition WL holds, by Lemma I.1, the event occurs with probability 1

Next we proceed to verify occurrence of . In the first iteration, the penalty loadings are defined as . By Assumption 6.1, Euniformly over . Moreover, Assumption 6.1(iv)(b) yields

with probability 1 . In turn this shows that for n large so that

with probability 1 for some uniformly bounded Moreover, ˜is uniformly bounded for n large enough which implies that as defined in (I.1) in Appendix I.2 is bounded away from zero with probability 1 by the condition on sparse eigenvalues of order (see Bickel et al. (2009) Lemma 4.1(ii)).

By Lemma I.3, since )] by the choice of

In the application of Lemma I.4, by Assumption 6.1(iv)(c), we have that minuniformly bounded for n large enough with probability 1 (1). Thus, with probability 1 by Lemma I.4 we have

for some ¯C independent of n, since uniformly in we have a sparsity bound and that ensures that a bound on the prediction rate yields a bound on the -norm rate through the relations

In the kth iteration, the penalty loadings are constructed based on (, defined as . We assume (satisfy the rates above uniformly in Then with probability 1 (1) we have uniformly in

where we used that maxby Assumption 6.1(iv)(a), and that inf2 with probability 1 large so that Further, for n large so that (2 ¯, this establishes that the event of the penalty loadings for the (k + 1)th iteration also satisfy for a uniformly bounded L and some with probability 1 (1) uniformly in

This leads to the stated rates of convergence and sparsity bound.

Proof of Theorem 6.2. In order to establish the result uniformly in , it suffices to establish the result under the probability measure induced by any sequence . In the proof we shall use P, suppressing the dependency of on the sample size n. The proof is similar to the proof of Theorem 6.1. We invoke Lemmas I.6, I.7 and I.8 which require Condition WL and some events to occur. We show that Assumption 6.2 implies Condition WL and that the required events occur with probability at least 1

Let denote the ideal penalty loadings, E]) the conditional variance of ) the rescaled approximation error as defined in (I.5). The three events required to occur with probability 1 follows: is large enough; L uniformly bounded, for the penalty loading in all iterations sufficiently large.

Regarding , by Assumption 6.2(iii), we have uniformly on , we have that the rescaled approximation error defined in (I.5) satisfies large enough so that . Assumption 6.2(iv)(b) yields supwith probability 1 occurs with probability 1

To apply Lemma I.1 to show that occurs with probability 1(1) we need to verify Condition WL. Condition WL(i) is implied by the sparsity in Assumption 6.2(i) and the covering condition in Assumption 6.2(ii). By Assumption 6.2 we have that is fixed and the Algorithm sets [1Assumption 6.2(i). Since it is assumed that Ein , Condition WL(ii) holds. Condition WL(iii) follows from Assumption 6.1(iv). Then, by Lemma I.1, the event occurs with probability 1

Next we verify the occurrence of . In the initial iteration, the penalty loadings are defined as . Assumption 6.2(iv)(c) for the sparse eigenvalues implies that for n large enough, with probability 1

Moreover, Assumption 6.2(iv)(b) yields

with probability 1 is bounded away from zero and from above uniformly over , with the same probability because E] is bounded away from zero and above. By (N.1) and Elarge enough, we have for some uniformly bounded with probability 1

Thus, ˜is uniformly bounded. In turn, since inf) is bounded away from zero, we have ¯definitions in (I.1) and (I.2). It follows that is bounded away from zero by the condition on sparse eigenvalues stated in Assumption 6.2(iv)(c), see Bickel et al. (2009) Lemma 4.1(ii).

By the choice of )]. By relation (I.4) and Assumption 6.2(iv)(a), inf. Under the condition the side condition in Lemma I.6 holds with probability 1 (1), and the lemma yields

In turn, under Assumption 6.2(iv)(c) and , with probability 1I.7 implies

for some ¯C independent of n, since by (N.16) we have uniformly in