Feedback Capacity and a Variant of the Kalman Filter with ARMA Gaussian Noises: Explicit Bounds and Feedback Coding Design

2020·arXiv

Abstract

In this paper, we relate a feedback channel with any finite-order autoregressive moving-average (ARMA) Gaussian noises to a variant of the Kalman filter. In light of this, we obtain relatively explicit lower bounds on the feedback capacity for such colored Gaussian noises, and the bounds are seen to be consistent with various existing results in the literature. Meanwhile, this variant of the Kalman filter also leads to explicit recursive coding schemes with clear structures to achieve the lower bounds. In general, our results provide an alternative perspective while pointing to potentially tighter bounds for the feedback capacity problem.

Index Terms—Feedback capacity, feedback channel, feedback coding, colored Gaussian noise, Kalman filter, Bode integral.

I. INTRODUCTION

THE feedback capacity [2] of additive colored Gaussiannoise channels has been a long-standing problem in information theory, generating numerous research results over the years, due to its interdisciplinary nature and significance in understanding and applying communication/coding with feedback. In general, we refer to the breakthrough paper [3] and the references therein for a rather complete literature review; see also [4], [5] for possibly complementary paper surveys. Meanwhile, papers on this topic have also been coming out continuously after [3], which include but are certainly not restricted to [1], [6]–[16]. More specifically, in [3], Kim proposed a variational characterization of the feedback capacity in terms of the noise power spectral density and showed that the feedback capacity can be achieved by linear feedback coding schemes, which, among the many other conclusions derived therein, yields for the first time an analytic expression when specialized to the first-order autoregressive moving-average (ARMA) noises. In [6], Ardestanizadeh and Franceschetti showed, from the perspective of Bode integral, that the feedback capacity is equal to the maximum instability that can be tolerated by any linear controller under a given power constraint. In [7], Liu and Elia established the mutual equivalences among the feedback capacity, the Cram´er-Rao bound or the minimum mean-square error in estimation systems, and the Bode integral in control systems. In [8], Stavrou

Song Fang and Quanyan Zhu are with the Department of Electrical and Computer Engineering, New York University, USA (e-mail: song.fang@nyu.edu; quanyan.zhu@nyu.edu).

This paper was presented in part at IEEE ISIT 2020 as [1], which, however, only considered the class of scalar plant parameters as A = a ∈ R and C = c ∈ R ; in other words, the main result therein is essentially Corollary 2 herein, a special case of what are presented in this paper. Note also that all the proofs in [1] were omitted due to lack of space.

et al. obtained sequential necessary and sufficient conditions in terms of input conditional distributions to achieve the feedback capacity. In [9], Liu and Han proved the uniqueness of the optimal solution to the variational characterization of [3] and proposed an algorithm to recursively compute the optimal solution with convergence guarantee, while, for any finite-order ARMA noises, providing a relatively more explicit formula as a simple function evaluated at a solution to a system of polynomial equations. In [10], Li and Elia showed that the problem of achieving feedback capacity coincides with the problem of finding stabilizing feedback controllers with maximal transmission rate over Youla parameters, and proposed an approach to numerically compute the feedback capacity while constructing feedback codes that are arbitrarily close to capacity-achieving. Subsequently in [11], Rawat et al. generalized the approach in [10] to the feedback capacity of multi-antenna channels with multivariate colored noises. In [12], Pedram and Tanaka analyzed structural properties of the optimal feedback policies and developed a convex program that can be used to compute the feedback capacity. In [13], Kourtellaris and Charalambous studied the feedback capacity problem by applying stochastic optimal control theory and a variational equality of directed information while developing a methodology to identify the information structures of optimal channel input conditional distributions. In [14], Gattami introduced a new approach to the feedback capacity problem by solving the problem over a finite number of transmissions and then taking the limit of an infinite number of transmissions. In [15], Ihara presented an alternative proof to the analytic expression for the first-order moving average (MA) noises. In [16], Aharoni et al. proposed a directed information estimation algorithm based on neural networks to compute the feedback capacity.

In particular, analytic or relatively explicit expressions or lower bounds, oftentimes in terms of a root of a polynomial equation, of the feedback capacity have been presented in [3], [4], [9], [15], [17]–[20]. More specifically, Butman obtained explicit lower bounds for the first-order autoregressive (AR) noises in [17] and then for any finite-order AR noises in [18]. In [19], Elia derived a refined, explicit lower bound for any finite-order AR noises. In [20], Yang et al. obtained an explicit lower bound for the first-order ARMA noises. For the first time, Kim discovered analytic formulae of feedback capacity, rather than its lower bounds, for the first-order AR noises in [4] and then for the first-order ARMA noises in [3]. In [9], Liu and Han discovered a relatively explicit expression for any finite-order ARMA noise in terms of a solution to a system of polynomial equations. In [15], Ihara presented an alternative proof to the analytic expression for the first-order MA noises derived in [4]. This line of work provided the main motivations for obtaining the results in this paper.

In general, this paper employs a control-theoretic approach to analyze feedback channels, which has been inspired by, e.g., [6], [7], [19] (Bode integral), [7], [12]–[14], [20], [21] (stochastic control and/or Kalman filter), as well as [10], [11] (Youla parametrization), in a broad sense; see also [22] and the references therein. One difference from the previous works, however, is that in this paper we adopt a particular variant of the Kalman filter that can deal with the first-order AR noises without extending the state to be estimated in Kalman filtering systems, as introduced in [23], and generalize the approach to cope with any finite-order ARMA noises. Another difference is that we examine the algebraic Riccati equation associated with the Kalman filter in a “non-recursive” characterization recently developed in [24]. Accordingly, we establish the connection between this variant of the Kalman filter and a feedback channel with any finite-order ARMA noises, after carrying out a series of equivalent transformations. In light of this connection, we obtain explicit lower bounds on the feedback capacity for any finite-order ARMA noises, by designing the parameters of the plant in the Kalman filtering system in a structural way. In addition, this variant of the Kalman filter naturally provides explicit recursive coding schemes with clear structures to achieve the lower bounds.

The lower bounds presented in this paper are seen to be consistent with various existing (analytic or explicit) results in the literature. Specifically, our bounds are shown to be tight for the first-order ARMA noises [3] (see also [9]) and a special class of the second-order MA noises [4]. Meanwhile, our results are seen to reduce to the lower bounds for any finite-order AR noises of [19] (see also [18]). Particularly, in comparison to the explicit expression of feedback capacity for any finite-order ARMA noises derived in [9] (see Theorem 18 therein), 1) what we obtained are lower bounds; 2) our explicit expression admits a simpler form; 3) it is not yet fully clear what the relationship between the two is. (Note that the approach taken in [9] was not from the perspective of Kalman filters.) In general, our results shall complement those of [9] in the sense that examining the gap between the expressions of [9] and ours might likely either simplify the expression in [9] or point to structures of the plant parameters with tighter bounds in our approach; in either case, additional insights will be gained into the feedback capacity problem. On the other hand, the explicit recursive coding schemes developed in this paper also complement the existing ones in the literature (see, e.g., [3], [19]).

The remainder of the paper is organized as follows. Section II provides technical preliminaries. Section III presents the main results. Concluding remarks are given in Section IV.

II. PRELIMINARIES

In this paper, we consider real-valued continuous random variables and discrete-time stochastic processes they compose. All random variables and stochastic processes are assumed to be zero-mean for simplicity and without loss of generality. We represent random variables using boldface letters. The logarithm is defined with base 2. A stochastic process {xk} is said to be asymptotically stationary if it is stationary as k → ∞ , and herein stationarity means strict stationarity [25]. Note in particular that, for simplicity and with abuse of notations, we utilize x ∈ R and x ∈ Rn to indicate that x is a real-valued random variable and that x is a real-valued n-dimensional random vector, respectively. Definitions and properties of the information-theoretic notions such as entropy rate h∞ (x) can be found in, e.g., [26].

A. Feedback Capacity

Consider an additive colored Gaussian noise channel with feedback given as

where {xk} , xk ∈ R denotes the channel input, {yk} , yk ∈ R denotes the channel output, {zk} , zk ∈ R denotes the additive noise that is assumed to be stationary colored Gaussian, and m denotes the message. The feedback capacity of such a channel with power constraint P is defined as [2]

where the supremum is taken over all channel input x1,...,k that satisfy

Recently in [3], it was discovered that (1) is equal to

where the supremum is taken over all stationary channel input processes {xk} of the form

while satisfying

In fact, [3] provided a sequence of equivalent characterizations of the feedback capacity, whereas for the purpose of this paper, it suffices to adopt the characterization of (2) and (3) herein (see the proof of Theorem 3).

B. Kalman Filter

We now give a brief review of (a special case of) the Kalman filter [23], [27]; note that hereinafter the notations are not to be confused with those in Section II-A. Particularly, consider

Fig. 1. The Kalman filtering system.

the Kalman filtering system depicted in Fig. 1, where the statespace model of the plant to be estimated is given by

Herein, xk ∈ Rn is the state to be estimated, yk ∈ R is the

plant output, and vk ∈ R is the measurement noise, whereas the process noise, normally denoted as {wk} [23], [27], is assumed to be absent. The system matrix is A ∈ Rn×n while the output matrix is C ∈ R1×n , and we assume that A is anti-stable (i.e., all the eigenvalues are unstable with magnitude greater than or equal to 1) while the pair (A, C) is observable (and thus detectable [28]). Suppose that {vk} is white Gaussian with variance σ2v = E�v2k�≥ 0 and the initial state is Gaussian with covariance E�x0xT0�≻ 0 . Furthermore, {vk} and are assumed to be uncorrelated. Correspondingly, the Kalman filter (in the observer form [28])

for (4) is given by

where xk ∈ Rn, yk ∈ R, ek ∈ R , and uk ∈ Rn . Herein, denotes the observer gain [28] (note that the observer gain is different from the Kalman gain by a factor of A; see, e.g., [23], [28] for more details) given by

where denotes the state estimation error covariance as

In addition, can be obtained iteratively by the Riccati equation

with P0 ≻ 0 . Additionally, it is known [23], [27] that since (A, C) is detectable, the Kalman filtering system converges,

Fig. 2. The steady-state Kalman filtering system in integrated form.

i.e., the state estimation error {xk − xk} is asymptotically stationary. Moreover, in steady state, the optimal state estimation error variance

attained by the Kalman filter is given by the (non-zero) positive semi-definite solution [23] to the algebraic Riccati equation

whereas the steady-state observer gain is given by

Meanwhile, it is known from [24] (by letting m = 1 and W = 0 in Theorem 1 therein; since the fact that {wk} is absent implicates W = E�wkwTk�= 0 ) that

since A is assumed to be anti-stable, whereas

since A is a real matrix.

yk − Cxk , we may integrate the systems of (4) and (5) in steady state into an equivalent form:

as depicted in Fig. 2, since all the sub-systems are linear.

III. FEEDBACK CAPACITY FOR ARMA GAUSSIAN NOISES

The approach we take in this paper to obtain lower bounds on the feedback capacity of channels with ARMA Gaussian noises is by establishing the connection between such feedback channels and a variant of the Kalman filter to deal with ARMA Gaussian noises. Towards this end, we first present the following variant of the Kalman filter.

A. A Variant of the Kalman Filter

Consider again the Kalman filtering system given in Fig. 1. Suppose that the plant to be estimated is still given by

only this time with an ARMA measurement noise {vk} , vk ∈ R, represented as

where {�vk} , �vk ∈ R is white Gaussian with variance σ2�v = E��v2k�> 0 . Equivalently, {vk} may be represented [29] as the output of a linear time-invariant (LTI) filter F (z) driven by input {�vk} , where

Herein, we assume that F (z) is stable and minimum-phase. We may then employ the method of dealing with colored noises without extending the state of the Kalman filter, as introduced in [23] (Chapter 11), after certain modifications. In fact, since therein the process noise {wk} is not absent, this approach is only applicable to the first-order AR noises; whereas in this paper, assuming that the process noise is absent, we may generalize the approach to any finite-order ARMA noises. Proposition 1: Denote

Then, (10) is equivalent to

where

Proof: Note first that since F (z) is stable and minimum-phase, the inverse filter

is also stable and minimum-phase. As a result, it holds ∀ |z| ≥ 1 that

i.e., the region of convergence must include, though not necessarily restricted to, |z| ≥ 1 . Consequently, for , we may expand

and thus {�vk} can be reconstructed from {vk} as [29]

Accordingly, it holds that

Meanwhile, since A is anti-stable (and thus invertible), we have xk−i = A−ixk . As a result,

Furthermore,

I −

On the other hand, we have shown that

|λℓ| ≥ 1 , ℓ = 1, . . . , n, we have

and hence

Therefore, it holds that

and thus

which completes the proof.

We shall now proceed to prove that the system given in (14) is observable (and thus detectable). For simplicity, we will denote (by a slight abuse of notation; cf. (16))

detectable).

is given by

As such, since (A, C) is observable, i.e., [28]

it suffices to show that F −1 (A) is invertible in order to show that

i.e., �A, �C� is also observable. To see this, note that it is known from the proof of Proposition 1 that

Fig. 3. The steady-state integrated Kalman filter for colored noises.

Then, since 1−�pi=1 fiz−i is minimum-phase while the poles of 1+�qj=1 gjz−j are given by zj = 0, j = 1, . . . , q , it holds for any ℓ = 1, 2, . . . , n that

due to the fact that |λℓ| > 1, ℓ = 1, . . . , n . Hence, F −1 (A) is invertible, and consequently, �A, �C� is observable (and thus detectable).

where xk ∈ Rn, yk ∈ R, ek ∈ R , and uk ∈ Rn . Furthermore, since �A, �C� is detectable, the Kalman filtering system converges, i.e., the state estimation error {xk − xk} is asymptotically stationary. Moreover, in steady state, the optimal state estimation error covariance

attained by the Kalman filter is given by the (non-zero) positive semi-definite solution to the algebraic Riccati equation

whereas the steady-state observer gain is given by

In addition, it holds that (cf. (6))

or equivalently,

yk − �Cxk , we may integrate the systems of (14) and (20) in steady state into an equivalent form:

as depicted in Fig. 3. In addition, it may be verified that the closed-loop system given in (24) and Fig. 3 is stable [23], [28].

B. Lower Bounds on Feedback Capacity and Feedback Coding

We now proceed to obtain lower bounds on feedback capacity as well as the corresponding recursive coding schemes to achieve them, based upon the results and discussions provided in the previous subsection. We first propose a particular way to design A and C.

Theorem 1: Suppose that

with a given γ ∈ R, γ ̸= 0 , has at least n ≥ 1 distinct nonminimum-phase zeros. Let

where λ1, λ2, . . . , λn, λ1 ̸= λ2 ̸= · · · ̸= λn , are picked among the nonminimum-phase zeros of (25). Note that conjugate zeros of (25), if there are any, should be picked in pairs in order to render A a real matrix. Note also that herein T ∈ Rn×n can be any invertible matrix. In addition, choose a C ∈ R1×n that renders (A, C) observable, e.g.,

Then, in the system of (24) and Fig. 3, it holds for any ℓ = 1, . . . , n that

Proof: Suppose that A and C are chosen as in (26) and (27), respectively. Clearly,

and it then follows from (19) that

As such,

Fig. 4. The steady-state integrated Kalman filter for colored noises: Equivalent form.

Meanwhile, note that the pair (A, C) is observable since the observability matrix is given by

and it can be verified that its rank is n, since

and λ1 ̸= λ2 ̸= · · · ̸= λn . Hence, (23) holds, which can then be rewritten as

since

Therefore, (28) follows by noting also (29).

Note in particular the fact that conjugate zeros of (25) are included in pairs implicates that

Note also that

since (25) has at most max {p, q} zeros.

Meanwhile, we may obtain an equivalent form of the system given in (24) and Fig. 3.

Theorem 2: The system in Fig. 3 is equivalent to that in Fig. 4, where K (z) is dynamic and is given as

Fig. 5. The steady-state integrated Kalman filter for colored noises: Equivalent form 2.

Fig. 6. The steady-state integrated Kalman filter for colored noises: Equivalent form 3.

Herein, is given by (21). More specifically, the system in

which is stable as a closed-loop system. Proof: Note first that the system in Fig. 3 is equivalent to the one in Fig. 5 since �K = F (z) K (z) . Then, since

we have

(Note that this “static-dynamic equivalence” transformation, where CF −1 (A) is static and F −1 (z) C is dynamic, is a critical step.) Consequently, the system of Fig. 5 is equivalent to that of Fig. 6. Moreover, since all the sub-systems are linear, the system of Fig. 6 is equivalent to that of Fig. 7, which in turn equals to the one of Fig. 4; note that herein F (z) is stable and minimum-phase, and thus there will be no issues caused by cancellations of unstable poles and nonminimum-phase zeros. Meanwhile, the closed-loop stability of the system given in (33) and Fig. 4 is the same as that of the system given by (24) and Fig. 3, since they are essentially the same feedback system.

Fig. 7. The steady-state integrated Kalman filter for colored noises: Equivalent form 4.

Fig. 8. The steady-state integrated Kalman filter for colored noises: Equivalent form 5.

Note in particular that in the system of (33) and Fig. 4, it holds that

since the system is LTI and {xk − xk} is stationary.

We next provide a key observation that enables relating the Kalman filtering system to the feedback capacity.

Observation 1: In the system of (33) and Fig. 4, or equivalently, in the system of Fig. 8, we may view

as a feedback channel [3], [6] with ARMA noise {vk} , whereas {−y′k} is the channel input while {e′k} is the channel output. Moreover, as illustrated in Fig. 8,

may be viewed as the feedback coding scheme; cf. [6], or alternatively, [3] with

Meanwhile, with the notations in (35), the feedback capacity is given by (cf. the definition in (2))

where the supremum is taken over all stationary channel input processes {−y′k} of the form

while satisfying

As such, if A and C are designed specifically as in Theorem 1, then (36) provides a class of sub-optimal feedback coding scheme as long as (40) is satisfied, and the corresponding

is thus a lower bound of (38).

Proof: Note that herein we have used the fact that {−y′k} is stationary and is a linear combination of the past {vk} up to time k − 1 , i.e., (39) holds. To see this, note first that, according to Theorem 2, the system in (33) and Fig. 4 is stable (and LTI). In particular, the transfer function from {vk} to , given as

is stable (and LTI). Then, since {vk} is stationary, {−y′k} is also stationary. On the other hand, it can be verified that (42) is strictly causal, since

is causal. As a result, (39) holds.

Based upon Observation 1, the following lower bound of feedback capacity can be obtained.

has at least one solution [λ1, . . . , λn]T that satisfies |λℓ| > 1, ℓ = 1, . . . , n while λ1 ̸= λ2 ̸= · · · ̸= λn and �nℓ=1 λℓ ∈ R ; note that herein ζ ∈ R, ζ ̸= 0 , is an auxiliary parameter. Among all such solutions, denote the one that provides the largest |�nℓ=1 λℓ| as �λ1, . . . , λn�T . Then, a lower bound of the feedback capacity with ARMA noise (11) and power constraint P is given by

Proof: Note first it is supposed that (43) has at least one solution [λ1, . . . , λn]T that satisfies |λℓ| > 1, ℓ = 1, . . . , n while λ1 ̸= λ2 ̸= · · · ̸= λn , which implicates that λ1, λ2, . . . , λn are in fact distinctive nonminimum-phase zeros of

In addition, the fact that �nℓ=1 λℓ ∈ R implicates that conju- gate zeros, if there are any, are included herein in pairs. We may then let

where T ∈ Rn×n can be any invertible matrix, and choose a C ∈ R1×n that renders (A, C) observable (see the proof of Theorem 1), e.g.,

Hence, in the system of (24) and Fig. 3, it holds for any ℓ = 1, . . . , n that (see Theorem 1)

On the other hand, it is known from (43) that

Thus,

which means that the power constraint of (40) is satisfied in the system of (33) and Fig. 4, since it is known from (34) that

As such, according to Observation 1, the corresponding (36) provides a class of sub-optimal feedback coding, whereas the corresponding

provides a lower bound of the feedback capacity. In addition, since the transfer function from {vk} to {e′k} is given by

it holds that (cf. discussions in [6], [19]),

where the first equality may be referred to [22], [25], the third equality follows as a result of the Bode integral or Jensen’s formula [22], [30], and the last equality since A is a real matrix (see (30)). Note that herein we have used the fact that F −1 (z) is stable and minimum-phase and

is stable. We have also utilized the fact that (A, C) is controllable, and thus the set of unstable poles of

is exactly the same as the set of eigenvalues of A with magnitudes greater than or equal to 1 [28]; see also, e.g., discussions in [24]. As such, the lower bound on feedback capacity is equivalently given by

Meanwhile, we may pick the solution �λ1, . . . , λn�T that denotes the one that provides the largest |�nℓ=1 λℓ| , that is, the largest log |�nℓ=1 λℓ| , among all the solutions of (43) that satisfy |λℓ| > 1, ℓ = 1, . . . , n while λ1 ̸= λ2 ̸= · · · ̸= λn and �nℓ=1 λℓ ∈ R . Accordingly, the lower bound on feedback capacity is given by

This completes the proof.

Equivalently, the lower bound can be rewritten, in terms of the “signal-to-noise” ratio P/σ2�v , as

since it holds for any ℓ = 1, . . . , n that (see the proof of Theorem 3)

whereas

In fact, (31) indicates that it is unnecessary to employ any n > max {p, q} in Theorem 3. Accordingly, we may obtain the following combined lower bound based on Theorem 3.

Corollary 1: A combined lower bound of the feedback capacity with ARMA noise (11) and power constraint P is given by

where λℓ, ℓ = 1, . . . , n , are given as in Theorem 3. We now show that the combined lower bound always exists by simply verifying the lower bound in Theorem 3 always exists for n = 1.

that

has at least one root that satisfies |λ| > 1, λ ∈ R . Among all such roots, denote the one with the absolute value as . Then, a lower bound of the feedback capacity with ARMA noise (11) and power constraint P is given by

which is equal to

In addition, the lower bound always exists, i.e., (48) does have at least one real root that satisfies |λ| > 1 .

to (48) and (49), respectively, whereas (45) becomes (50). To show that (48) does have at least one real root that satisfies |λ| > 1 , note first that when λ = 1 ,

whereas

Herein, we have used the fact that

is minimum-phase, and hence for z = 1,

In addition, since

is continuous on (1, ∞) , it follows that (48) has at least one root within , i.e., a real root with |λ| > 1 .

In particular, for AR noises, i.e., when gj = 0, j = 1, . . . , q , the lower bound in Corollary 2 reduces to the lower bound of [19] (see Section V.B therein), which also discussed the relation to the previous lower bound obtained in [18].

We next consider the special case of first-order ARMA noises with p = q = 1. In this case, since max {p, q} = 1, it suffices to consider only n = 1, and accordingly, Example 1 follows directly from Corollary 2.

Example 1: In particular, when p = q = 1, a lower bound of the feedback capacity with the first-order ARMA noise

assuming that

is stable and minimum-phase, is given by

with the largest absolute value; note that it has been proved in Corollary 2 that (52) has at least one real root with absolute value greater than 1. Moreover, the lower bound may be rewritten as

where denotes the largest positive real root to be chosen among the roots of

Example 1 is consistent with the feedback capacity of such channels derived in [3] (see also [9]). In fact, it has been shown in [3] (see Theorem 5.3 therein) that the feedback capacity, rather than its lower bound, is given by the unique (therefore, the largest) positive real root of

when f1 + g1 ≤ 0 , while when f1 + g1 > 0 , the feedback capacity is given by the only positive real root of

Note that it may be verified that when f1+g1 ≤ 0 , the positive real root of (56) is larger than or equal to that of (57), and vice versa. That is to say, the lower bound in Example 1 is indeed tight. (It is also worth mentioning that before [3], lower bounds on feedback capacity for the first-order AR noises and the first-order ARMA noises have been respectively obtained in [17], [18] and [20]. Meanwhile, see [4], [15] for further discussions on the feedback capacity for the first-order MA noises.)

We next consider the special case of second-order MA noises.

Example 2: By letting p = 0 and q = 2, we may now obtain lower bounds on the feedback capacity with the second-order MA noise

assuming that 1+g2z−2 is minimum-phase. In addition, since max {p, q} = 2, it suffices to consider n = 1 and n = 2.

• Case 1: n = 1. In this case, Corollary 2 will give the following lower bound:

with the largest absolute value. According to [4] (see Section IV therein), this lower bound is not tight.

with the largest absolute value. Correspondingly, the lower bound becomes

This lower bound is indeed tight, i.e., (63) is the feedback capacity of the channel given in (58); see discussions in [4]. Meanwhile, (63) is also equivalent to

with the largest absolute value. As a matter of fact, (64) is the same as the feedback capacity with the first-order MA noise [4]

Meanwhile, (33) and Fig. 8 essentially provide a recursive coding scheme/algorithm to achieve the lower bound in Theorem 3; see also discussions in Observation 1.

Theorem 4: One class of recursive coding scheme to achieve the lower bound in Theorem 3 is given by

where

and

while can be found by (21). Herein, T ∈ Rn×n can be any invertible matrix. Note in particular that −y′0 represents the message (cf. discussions in [3]). We next provide two examples on the recursive coding schemes to achieve the feedback capacity in Example 1 and Example 2, respectively. Example 3: One class of recursive coding scheme to achieve the feedback capacity in Example 1 is given by

where c can be any non-zero real number. Correspondingly, (36) reduces to

while (21) becomes

since in this case

Furthermore, one might also compare (69) and (70) respectively with the coding scheme and B (z) for the first-order ARMA noises as presented in [3] by noting that

Example 4: One class of recursive coding scheme to achieve the feedback capacity in Example 2 (as in Case 2 therein) is given by

where

where T −1 ∈ R2×2 can be any invertible matrix. Correspondingly, (36) reduces to

where can be found by (21).

IV. CONCLUSION

In this paper, from the perspective of a variant of the Kalman filter, we have obtained explicit lower bounds on the feedback capacity of channels with any finite-order ARMA Gaussian noises, as well as the accompanying recursive coding schemes to achieve them. Potential future research problems include investigating the tightness of the lower bounds beyond the cases considered in this paper. It is also worth mentioning that the results presented in this paper represent the relatively preliminary explorations under the current framework; particularly, we considered a very special class of the plant parameters A and C with a simple structure. For future research, it might be interesting to investigate further the structure of the plant parameters, to make use of the yet unexploited degrees of freedom therein, which may potentially lead to tighter bounds and gain additional insights into the feedback capacity problem. For another possible future research direction, examining the gap between the expressions of [9] and ours might likely either simplify the expression in [9] or point to structures of the plant parameters with tighter bounds in our approach.

REFERENCES

[1] S. Fang and Q. Zhu, “A connection between feedback capacity and Kalman filter for colored Gaussian noises,” in Proceeedings of the IEEE International Symposium on Information Theory (ISIT), 2020, pp. 2055– 2060.

[2] T. M. Cover and S. Pombra, “Gaussian feedback capacity,” IEEE Transactions on Information Theory, vol. 35, no. 1, pp. 37–43, 1989.

[3] Y.-H. Kim, “Feedback capacity of stationary Gaussian channels,” IEEE Transactions on Information Theory, vol. 56, no. 1, pp. 57–85, 2010.

[4] ——, “Feedback capacity of the first-order moving average Gaussian channel,” IEEE Transactions on Information Theory, vol. 52, no. 7, pp. 3063–3079, 2006.

[5] ——, “Gaussian feedback capacity,” Ph.D. dissertation, Stanford Uni- versity, 2006.

[6] E. Ardestanizadeh and M. Franceschetti, “Control-theoretic approach to communication with feedback,” IEEE Transactions on Automatic Control, vol. 57, no. 10, pp. 2576–2587, 2012.

[7] J. Liu and N. Elia, “Convergence of fundamental limitations in feedback communication, estimation, and feedback control over Gaussian channels,” Communications in Information and Systems, vol. 14, no. 3, pp. 161–211, 2014.

[8] P. A. Stavrou, C. D. Charalambous, and C. K. Kourtellaris, “Sequential necessary and sufficient conditions for capacity achieving distributions of channels with memory and feedback,” IEEE Transactions on Information Theory, vol. 63, no. 11, pp. 7095–7115, 2017.

[9] T. Liu and G. Han, “Feedback capacity of stationary Gaussian channels further examined,” IEEE Transactions on Information Theory, vol. 65, no. 4, pp. 2492–2506, 2018.

[10] C. Li and N. Elia, “Youla coding and computation of Gaussian feedback capacity,” IEEE Transactions on Information Theory, vol. 64, no. 4, pp. 3197–3215, 2018.

[11] A. Rawat, N. Elia, and C. Li, “Computation of feedback capacity of single user multi-antenna stationary Gaussian channel,” in Proceedings of the Annual Allerton Conference on Communication, Control, and Computing, 2018, pp. 1128–1135.

[12] A. R. Pedram and T. Tanaka, “Some results on the computation of feedback capacity of Gaussian channels with memory,” in Proceedings of the Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2018, pp. 919–926.

[13] C. K. Kourtellaris and C. D. Charalambous, “Information structures of capacity achieving distributions for feedback channels with memory and transmission cost: Stochastic optimal control & variational equalities,” IEEE Transactions on Information Theory, vol. 64, no. 7, pp. 4962– 4992, 2018.

[14] A. Gattami, “Feedback capacity of Gaussian channels revisited,” IEEE Transactions on Information Theory, vol. 65, no. 3, pp. 1948–1960, 2018.

[15] S. Ihara, “On the feedback capacity of the first-order moving average Gaussian channel,” Japanese Journal of Statistics and Data Science, pp. 1–16, 2019.

[16] Z. Aharoni, D. Tsur, Z. Goldfeld, and H. H. Permuter, “Capacity of continuous channels with memory via directed information neural estimator,” in Proceedings of the IEEE International Symposium on Information Theory, 2020, pp. 2014–2019.

[17] S. Butman, “A general formulation of linear feedback communication systems with solutions,” IEEE Transactions on Information Theory, vol. 15, no. 3, pp. 392–400, 1969.

[18] ——, “Linear feedback rate bounds for regressive channels,” IEEE Transactions on Information Theory, vol. 22, no. 3, pp. 363–366, 1976.

[19] N. Elia, “When Bode meets Shannon: Control-oriented feedback com- munication schemes,” IEEE Transactions on Automatic Control, vol. 49, no. 9, pp. 1477–1488, 2004.

[20] S. Yang, A. Kavcic, and S. Tatikonda, “On the feedback capacity of power-constrained Gaussian noise channels with memory,” IEEE Transactions on Information Theory, vol. 53, no. 3, pp. 929–954, 2007.

[21] S. Tatikonda and S. Mitter, “The capacity of channels with feedback,” IEEE Transactions on Information Theory, vol. 55, no. 1, pp. 323–349, 2009.

[22] S. Fang, J. Chen, and H. Ishii, Towards Integrating Control and Information Theories: From Information-Theoretic Measures to Control Performance Limitations. Springer, 2017.

[23] B. D. O. Anderson and J. B. Moore, Optimal Filtering. Prentice-Hall, 1979.

[24] S. Fang, H. Ishii, and J. Chen, “An integral characterization of optimal error covariance by Kalman filtering,” in Proceedings of the American Control Conference, 2018, pp. 5031–5036.

[25] A. Papoulis and S. U. Pillai, Probability, Random Variables and Stochastic Processes. New York: McGraw-Hill, 2002.

[26] T. M. Cover and J. A. Thomas, Elements of Information Theory. John Wiley & Sons, 2006.

[27] T. Kailath, A. H. Sayed, and B. Hassibi, Linear Estimation. Prentice Hall, 2000.

[28] K. J. ˚Astr¨om and R. M. Murray, Feedback Systems: An Introduction for Scientists and Engineers. Princeton University Press, 2010.

[29] P. P. Vaidyanathan, The Theory of Linear Prediction. Morgan & Claypool Publishers, 2007.

[30] M. M. Seron, J. H. Braslavsky, and G. C. Goodwin, Fundamental Limitations in Filtering and Control. Springer, 1997.

Designed for Accessibility and to further Open Science