CONSIDER an agent traversing a directed graph whosenodes, upon being visited, emit a color taken from a set of possibilities. We are interested in identifying the nodes visited by the agent based on the colors observed. If there are probabilities associated with the edges and color emissions then the system described is a hidden Markov model (HMM) [1], otherwise it is a weak model [2]. Weak models are used in a variety of applications including target tracking in sensor networks [2] and cybersecurity [3].

This paper concerns itself only with the reconstruction of the node sequence given the observed color sequence. We assume that the graph and its coloring are known a priori, as we are not studying the more challenging task of inferring the graph and its coloring from observed sequences of colors [4].

In general, no guarantees can be made about the ability to reconstruct the sequence of nodes visited. In our previous work, we laid out a taxonomy of observability classes for which different guarantees can be made [5]. In particular, trackable models are weak models for which the number of node sequences (the “hypotheses”) consistent with an observed color sequence is bounded by a polynomial in the sequence length [2]. For models which are not trackable, the growth rate is exponential. While polynomial growth is certainly better than exponential growth, a trackable model can still give rise to an unbounded number of hypotheses, which suggests that trackability is not a sufficient condition for accurate reconstruction of the node sequence.

In this paper, however, we present several new results which show that the node sequence reconstruction problem for trackable models which are strongly connected is much more feasible than in the more general case with transient nodes. As strong connectivity is a reasonable assumption when observing the steady state behavior of a system, these results suggest that the combination of trackability with strong connectivity can be sufficient to enable accurate reconstruction of the node sequence. This has important implications for the design of sensor networks and tracking systems, as trackability imposes much weaker constraints on the model’s coloring than the observability classes with stronger guarantees such as unifilar [6] and observable [7].

The structure of this paper is as follows: after establishing the necessary background and notation in Section 2, we show in Section 3 that the number of hypotheses for strongly-connected trackable models is bounded by a constant. Then in Section 4 we show that, for such models, it is always eventually possible to reconstruct which branch was taken at a point where there are multiple out-neighbors capable of emitting the same color. We illustrate how to quantify what “eventually” means using the mean recurrence time and find that the results agree well with a simulation. Section 5 provides an upper bound on the size of the hypothesis set for strongly-connected trackable models. Section 6 presents new results about the entropy rates of possible node sequences given observation sequences. Section 7 summarizes these results.

A weak model consists of a set of nodes V (the states a system can transition between), a set of edges E consisting of ordered pairs of nodes (the allowed state

Fig. 1: Transformation of a multi-colored model into a single-colored model. Node b can emit either or , so it is split into nodes bb and br, respectively. (This paper will use both color and shape to distinguish the “colors” which can be emitted by a given node.)

transitions), a set of possible colors (called the “symbols” in other work on weak models) which can be emitted, and a mapping which indicates which subset of can be emitted by a given node [2]. This can be seen as a node-multi-colored directed graph, and is a non-probabilistic version of a hidden Markov model (HMM) with discrete symbols [1]. We restrict our attention to graphs with a finite number of nodes.

Any multi-colored model can be transformed to an equivalent single-colored model (i.e., a model for which ) by replacing each multi-colored node with multiple nodes, one for each color, as shown in Figure 1. We will therefore assume, without loss of generality, that all models in this paper are single-colored. In this case, we can replace L with the function , which is equivalent to the lumping function in [8], [9]. An HMM with single-colored nodes is also known as a lumped Markov chain.

Consider the problem of tracking a system described by such a model. The structure of the graph is known, as is the sequence of colors emitted as the system transitions between the various states. The goal is to reconstruct the sequence of nodes visited. In general, no guarantees can be made about the accuracy of this inference problem. As mentioned above, however, there is a specific class of weak models called trackable weak models for which the number of hypotheses consistent with an observed color sequence is bounded by a polynomial in the sequence length [2]. Formally, let be the set of node sequences which are consistent with the color sequence . We refer to any such node sequence as a hypothesis. Let be the worst-case number of hypotheses for a color sequence of length t. A trackable model is defined to be a model for which for some . Models which are not trackable must have exponential hypothesis growth rates: for some c > 0.

The original work on trackable models characterizes trackability both in terms of the joint spectral radius of the transition matrices defining the model as well as by considering the node sequences which are consistent with all possible color sequences. As we describe in another publication, trackable models are also characterized by the absence of intersecting cycles which permit the same color sequence [5]. Two cycles indexed by i are said to be intersecting if there is at least one i such that and are said to permit the same color sequence

Fig. 2: Basic examples of models which are (a) untrackable and (b) trackable.

Examples of trackable and untrackable models are given in Figure 2. In the model shown in Figure 2a, the two intersecting cycles of the form ( ,

, ) cause the model to be untrackable, because this color sequence begins and ends at the central node but permits two paths – the model lacks the “unique path property” defined in [2]. Indeed, every time the color sequence ( ,

, ) is observed the number of possible node sequences doubles, resulting in the exponential growth expected for an untrackable model. The model in Figure 2b, on the other hand, does not possess this pathology. While it may not be possible to know what node the system is at for arbitrarily long periods of time (e.g., when a color sequence of the form ( , , . . . ) is observed), there is no color sequence which causes exponential growth in the number of hypotheses. In fact, the results of Section 3 show that the number of hypotheses is O(1) for this model – at any given time, there are only one or two possible node sequences which are consistent with the observed color sequence.

It has been shown that for a trackable model with k > 0 (unbounded polynomial growth) or O(1) (bounded growth) are the only two possible cases [2]. For unbounded growth to occur there must exist three paths such that:

1) All three paths permit the same color sequence: 2) is a cycle of period

3) is a cycle of period T which is distinct from :

4) is a path of length T + 1 which starts at the beginning of and ends at the beginning of :

This structure is shown for an abstract model in Figure 3. The “branch” contains whichever nodes and edges in the path from to are not contained in the cycles and .

Suppose now that the model is strongly connected, meaning that every node is reachable from every other node. For this condition to hold, there must be at least one “return path” to get from back to . Using this structure, we can prove the following theorem.

Fig. 3: Components of a strongly connected model with paths , and . The “branch” is the portion of which is part of neither or . The “return path” must exist for the model to be strongly connected. This construct is relevant to Theorem 1.

TABLE 1: Cycles and path associated with Figure 4.

Theorem 1. In a strongly-connected trackable model O(1).

Proof. Suppose a strongly-connected trackable model G has with k > 0, and hence has the three paths , described above. In such a model, it is always possible to construct a pair of intersecting cycles having the same color sequence according to the following recipe:

1) and start at a node in which connects to a return path.

2) and both traverse the return path and begin to follow .

3) and traverse until the branch to is reached.

4) stays on while follows then begins to follow .

5) reaches the branch a second time and follows while continues to follow .

6) and follow until they reach the starting node in , thus completing the cycles.

Note that it does not matter how long the return path is, if there are multiple return paths, or at what point the return path connects between and , because and traverse a given return path together. Also, because has the same length as and and will reconnect at the same node in . Because a trackable model cannot have intersecting cycles which permit the same color sequence, this contradicts our assumptions. Therefore, a strongly-connected trackable model cannot have , and and hence must have .

To illustrate this proof, consider the model shown in Figure 4. This model has and as listed in Table 1. Following the recipe from the proof, we can construct the pair of cycles given in Table 2. The color sequence given in the table permits two paths which begin and end at the same node, which means the size of the hypothesis set will double each time the color sequence is observed – this exponential

branch

Fig. 4: Strongly-connected model with , and . To make the example more general, the branch and return path were constructed to consist of more than a single edge.

growth in the number of hypotheses means that the model is not trackable.

This theorem has good implications for the feasibility of the node sequence reconstruction problem in strongly-connected trackable models. Specifically, the basic implication of trackability is that there is a possibility for unbounded polynomial growth – a tracking system with finite memory monitoring a trackable model for a long time would require a mechanism for pruning unlikely but permissible hypotheses. But if a trackable model is also strongly connected we only ever have to keep track of a finite number of hypotheses, regardless of the observation length.

In a trackable model which is not strongly connected, we can discuss the implications of this theorem by borrowing some terminology from the study of Markov chains [10]. The set of nodes which can be reached from node i are called the nodes accessible from i. A node i is called recurrent if, for every node j which is accessible from i, i is accessible from j. A node which is not recurrent is called transient. The set of nodes reachable from a recurrent node form a recurrent class. Note that a model could have multiple disjoint recurrent classes.

Using this terminology, we can state the following corollary to Theorem 1.

Corollary 2. In a trackable model G for which with k > 0, the cycle involves only transient nodes.

Proof. For to hold, the paths , and described above must exist. But, for the model to be trackable, there can be no return path from the nodes in back to the nodes in , otherwise the recipe in the proof to Theorem 1 would lead to a contradiction. Therefore, the nodes in cannot be accessible from the nodes in . But, the nodes in are accessible from the nodes in , so the nodes in are transient.

The behavior of a weak model or a Markov chain can be divided into two parts: the burn-in period when it visits the transient nodes, and the steady-state period when it has entered a recurrent class. Corollary 2 implies that unbounded polynomial growth arises only from the transient burn-in period. It may or may not be possible, however, to tell when the system has transitioned between burn-in and steady-state behavior. Consider the model in Figure 5a: it is never possible to tell when the system has transitioned from a to b, and hence increases linearly in t. In the model in

TABLE 2: Cycles and associated with Figure 4.

Fig. 5: Model for which and it (a) is not and (b) is eventually possible to tell that the system has entered a recurrent class.

Figure 5b, however, once is observed we know the system is at c and hence has entered the recurrent class {b, c} and will stay there – for this example, remains fixed at whatever size it was before was observed.

In cases where it is not possible to know when the system has entered a recurrent class, it is necessary to characterize the typical length of the burn-in period so that unlikely trajectories which spend excessive time in transient nodes may be removed from the hypothesis set. In a Markov chain the expected number of time steps before a recurrent class is entered when starting from state i is called the mean absorption time , and is found by solving the following system of equations [10]:

where is the probability of transitioning from node i to node j. For the model in Figure 5a, we have

For example, if then : we expect to have reached b after 10 steps or so, after which we can remove the highly unlikely trajectories of the form (a, a, . . . ) from the hypothesis set.

The hypothesis set grows whenever a node which has same-colored out-neighbors is encountered. Formally, given , we say that and are same-colored out-neighbors of v if .2 If there are m out-neighbors of v such that is the same for all i, we refer to m as the multiplicity of the same-colored out-neighbors. If a system starts out at a known node, the nodes

visited can be reconstructed unambiguously until a same-colored out-neighbor is encountered. Once a same-colored out-neighbor is encountered the hypothesis set branches, and a key question is whether or not we can eventually determine which branch was taken. The following theorem shows that this reconstruction is always possible.

Theorem 3. In a strongly-connected trackable model, it will eventually be possible to identify which branch was taken at a node that has same-colored out-neighbors.

Proof. Suppose the theorem does not hold. Because the model is assumed to be strongly connected and trackable, a node v with same-colored out-neighbors of multiplicity m can be revisited an infinite number of times during an observation of infinite length. Every time the same-colored out-neighbors of v are encountered, the number of hypotheses which go through v at that time is multiplied by m. If it were not possible to identify which branch was taken before the system encounters the same-colored out-neighbors again, the number of such hypotheses would not have shrunk back down before this multiplication occurs. This would lead to exponential growth in the size of the hypothesis set, contradicting the assumption that the model is trackable.

This has good implications for reconstruction of the sequence of nodes visited in a strongly-connected trackable model. Specifically, if it is possible to uniquely identify the node the system visits at some time (such as when the system starts at a known node), then it will eventually be possible to uniquely identify all subsequent nodes visited by the system.

To quantify what “eventually” means in a Markov chain, consider a node v which has same-colored out-neighbors. We know that the question of which branch was taken must be resolved by the time the system transitions out of v a second time. (It may of course be much quicker, depending on the specific structure of the model – see below for further discussion of this point.) The expected time to travel from a node v back to itself is called the mean recurrence time . This can be computed by first computing the mean first passage times (i.e., the expected time to first visit v when starting from i) by solving the following system of equations [10]:

then substituting the result into

(Note that the calculation of mean first passage times is very similar to the calculation of mean absorption times in the

Fig. 6: (a) Model to illustrate the mean time to reconstruct which branch out of a is taken. The transition probabilities used for the example are indicated on each edge. The nodes are single-colored, so the emission matrix needed to express this as a discrete-symbol hidden Markov model only has one non-zero element in each row. (b) Error probability as a function of lag . The results (blue solid) are shown on a log-linear scale together with the best fit line (orange dotted). The behavior is well-described by an exponential with a time constant of 19.

previous section. Mean absorption times are the expected time until the first passage of any recurrent node, whereas mean first passage times are the expected time before the first passage of the specific node v.)

As an example, consider the model shown in Figure 6a. Once the system leaves a, it could spend an arbitrary amount of time in the cycles of the form ( ,

, ). But, when the system returns to a, either the sequence ( ,

, ) is seen, which indicates whether the system took the left- or right-hand branch, respectively. For the model in Figure 6a,

In order to illustrate this, we simulated 10 000 traversals, each consisting of 200 time steps. (We used many traversals in order to obtain a good estimate the error rate as a function of lag.) In order to avoid issues with reconstruction of the first couple nodes, we started each traversal at a. We reconstructed the most likely node sequence using the Viterbi algorithm and computed the accuracy as a function of the lag , where and are the estimated and true nodes at time t, respectively. The results are shown in Figure 6b. The behavior of the error probability is linear when shown on a log-linear scale, indicating that it is

described by a function of the form

where is the time constant and A is the error probability at . The best fit values are : the time constant of the recovery in accuracy following a same-colored out-neighbor is roughly equal to the mean recurrence time, as expected. Furthermore, the accuracy for is , consistent with the 50/50 chance of being in the left-hand or right-hand branch.

Note, however, that the mean recurrence time can be a very pessimistic estimate of the time needed to determine which branch was taken for two reasons. First, the structure of the model may permit unique reconstruction of which branch was taken well before the same-colored out-neighbors are revisited. If, for example, node c in the present example were changed to emit , the ambiguity caused by the color sequence ( , ) would be resolved on the very next step when either or is observed. Even if the structure of the model does not permit early reconstruction in this manner, however, the transition dynamics themselves may facilitate better performance than would be expected from the mean recurrence time alone. For example, suppose that is reduced to 0.01 in the present example. This yields , but repeating the above analysis gives . This is because the average time spent in the two branches is now dramatically different, so sufficiently long sequences of the form ( ,

, , . . . ) are correctly inferred to have come from the right-hand branch even before node a is revisited.

Having established that strongly-connected trackable models have a bounded number of hypotheses, it is of interest to know what that bound is. The main theorem of this section stems from the observation that the size of the hypothesis set only increases when:

1) the first color observed can be emitted by more than one node, or

2) a same-colored out-neighbor is encountered (as discussed in the previous section).

Before stating and proving the theorem, we need the following corollary to Theorem 3.

Corollary 4. In a strongly-connected trackable model for which observations start from a known node, there can be at most one hypothesis which visits a given node at a given time.

Proof. Suppose the corollary does not hold. Consider a color sequence such that, at some point after the known starting point, multiple hypotheses are created because of same-colored out-neighbors. Furthermore, suppose that the color sequence permits a pair of these hypotheses to visit the same node at the same time. Because the subsequent transitions do not depend on any of the previous transitions (Markov property), this means that which branch was taken at the node with same-colored out-neighbors can never be reconstructed, contradicting Theorem 3.

As an alternate proof of the corollary, note that color sequence described above would permit the construction of

Fig. 7: This model illustrates the bounds developed in Theorem 5.

two paths from the (known) starting node back to itself. Therefore, a model which permits such a color sequence lacks the unique path property, violating the assumption that the model is trackable.

Note that a given node may have multiple sets of same-colored out-neighbors. Let be the maximum multiplicity of the sets of same-colored out-neighbors at node v, where for a node which lacks same-colored out-neighbors. As an example, for the model in Figure 6a, and .

In addition, let K to be the largest number of same colored nodes in the model. For example, for the model in Figure 6a, K = 4.

Theorem 5. In a strongly-connected trackable model, G,

When the observations start from a known node,

Proof. We start with the case in which the observations begin at a known node. Consider a color sequence with the following properties:

1) Every node which has same-colored out-neighbors is visited by a path consistent with the color sequence.

2) The paths consistent with the color sequence include transitions into the highest-multiplicity set of same-colored out-neighbors at each node.

3) The paths consistent with the color sequence visit every node with same-colored out-neighbors on the path before the reconstruction promised by Theorem 3 occurs for any of the same-colored out-neighbors encountered.

Clearly this color sequence will generate the largest hypothesis set possible. If such a color sequence does not exist, then the maximum will be smaller than the bound obtained by assuming such a color sequence exists, thus accounting for the less-than part of the theorem.

To obtain the equal-to part, assume that such a color sequence exists and keep a running tally of as it is processed. When starting from a known node, . At each same-colored out-neighbor, the number of hypotheses through the node with same-colored out-neighbors is multiplied by . Corollary 4 means that there is only ever a single hypothesis going into the node with same-colored out-neighbors at a given time, so we add to our running tally to replace the single hypothesis with the new hypotheses. Because for nodes with no same-colored out-neighbors, we can simply sum over all nodes to obtain the expression given.

Returning to the case in which the observations do not begin at a known node, note that there are at most K nodes with the color first observed and, for each of those possible starting nodes, there are at most hypotheses as we have just established. As a result, there are at most hypotheses even with the ambiguity introduced by not knowing the node when the tracking began.

The difference between knowing the starting node and not knowing the starting node is illustrated concretely using the model in Figure 7. Note that nodes can have multiple same-colored in-neighbors. If observations start when the system is at an unknown same-colored in-neighbor, it will never be possible to identify the first node that was visited. This would cause to be multiplied by the number of hypotheses generated by this initial ambiguity. For the model in Figure 7, Theorem 5 indicates that as long as observations start at a known node. This bound is realized by the color sequence ( ,

, ). But, suppose instead that observations start at an unknown one of the same-colored in-neighbors of a (i.e., one of {q, r, s, t, u}). This results in the color sequence ( ,

, ), which in turn yields because it is not possible to determine which of the nodes was visited initially. Note that the unknown starting point bound is only tight if the color sequence used in the proof of Theorem 5 exists and the initial ambiguity from starting at one of the K same-colored nodes is never resolved. For example, the bound of is not tight for the model in Figure 7 because the elevenfold ambiguity in an initial observation of

is resolved by the time any of the non- nodes are visited. In general, there are several circumstances under which the initial ambiguity may not be resolved and the bound can be tight:

1) The K same-colored nodes could be same-colored in-neighbors of some node.

2) There could be a color sequence which permits paths from each of the K same-colored nodes to some node. This is a multi-step generalization of the same-colored in-neighbors in the previous case.

3) The model could be symmetric (i.e., |Aut(G)| > 1, where Aut(G) is the automorphism group of

model3 G): if the K same-colored nodes are not fixed points of the automorphisms then it will not be possible to resolve the initial ambiguity.

As an example of the last case, the model in Figure 8 has |Aut(G)| = K = 2 and no same-colored out-neighbors, so the bound of is tight.

This section considers the conditional entropy rate of the node sequence given the color sequence. As entropy rates are determined by the transition and emission probabilities, we consider any given weak model in this section to have been converted to a lumped Markov chain through assignment of transition probabilities such that . We can then speak of the conditional entropy rate of a weak model by establishing inequalities which hold for any assignment of probabilities which satisfies this condition.

The node sequence, , is a stationary, Markovian stochastic process. The color sequence, , is also a stationary stochastic process which may or may not be Markovian [8], [9]. The entropy rate of stochastic process conditional on stochastic process is

where is the classical Shannon conditional entropy for possible realizations of and [11]. Specifically,

where and are the sample spaces of the random variables and , respectively. The conditional entropy rate corresponds to the average amount of information about which is lost per time step when only is observed.

Note that trackability is equivalent to having an infi-nite “split-merge index” as defined in the recent work on lumped Markov chains by Geiger and Temmel [9]: the nonexistence of differing realizable trajectories with the same start and end points is, for the irreducible case considered by Geiger and Temmel, equivalent to the unique path property which characterizes trackability. Theorem 1 of [9] shows that the conditional entropy rate H(X|Y ) of the state sequence process conditioned on the observation sequence is strictly positive for a lumped Markov chain if and only if the corresponding weak model is untrackable. In other words, for an untrackable model there is always positive uncertainty about the process’ state however long the observation sequence happens to be.

Moreover, Geiger and Temmel also show that if there are only a bounded number of hypotheses per observation sequence, then the conditional entropy rate is H(X|Y ) = 0. Because their work assumes that the underlying Markov chain is irreducible (equivalent to our constraint of strong connectivity), they are silent about the case when the hypothesis growth is unbounded polynomial. As shown above, a weak model that is trackable but has unbounded worst case hypothesis growth cannot be strongly connected/irreducible. The following theorem addresses this case.

Theorem 6. A model which has (potentially unbounded) polynomial growth in the hypothesis set, with , has H(X|Y ) = 0.

Proof. Recall Equation (10). If the cardinality of is bounded by a polynomial for every observation sequence, then the largest entropy

occurs for the uniform distribution over elements so that

Consequently,

Thus, if hypothesis growth is bounded by a polynomial then H(X|Y ) = 0.

To gain some intuition for the implications of this theorem, note that there are models for which H(X|Y ) = 0 but it is still impossible to uniquely reconstruct X from Y . An example of a strongly-connected model for which this is the case is shown in Figure 8. Because it is trackable, this model has H(X|Y ) = 0. But if observations do not start at a known node the hypothesis set always has size two. An intuitive way of thinking about this case is that, assuming all nodes are equally likely starting points, which of the two hypotheses is correct represents one bit of uncertainty: , independent of T. Because this uncertainty does not grow as , it is taken to zero by the 1/T term inside the limit of Equation (9).

For a less obvious case, Theorem 6 shows that the model in Figure 5a has H(X|Y ) = 0 despite the fact that there is unbounded growth in the number of hypotheses. An intuitive explanation of the fact that H(X|Y ) = 0 despite

Fig. 8: Model from Figure 4 of [9]. The model is trackable and hence the conditional entropy rate is H(X|Y ) = 0 but it is not possible to uniquely reconstruct X from Y without additional constraints on the distribution of initial states.

the growth in uncertainty at each step is that the number of hypotheses (and hence the conditional entropy) grows at a rate which is sufficiently slow that the 1/T term can still take the conditional entropy rate to zero. In contrast to this case, Theorem 1 of [9] shows that the exponential growth in the number of hypotheses for untrackable models is fast enough to result in a nonzero entropy rate.

Note that for k = 0 (bounded growth) this theorem is more general than Theorem 1 of [9] because it applies even if the model is not strongly connected and/or has periodic nodes. That being said, the theorem is not an “if and only if” statement: H(X|Y ) = 0 does not imply that is bounded by a polynomial, which is what the following theorem addresses.

Theorem 7. A model for which all recurrent nodes are aperiodic has H(X|Y ) = 0 if and only if all of the recurrent classes are trackable.

Proof. Note that the limit in the definition of the conditional entropy rate means that H(X|Y ) depends on the long-term, asymptotic behavior of a model. Therefore, the transient behavior of the model is irrelevant in determining the conditional entropy rate: while the transient nodes can generate a color sequence which makes the hypothesis set arbitrarily large, we know that, with probability one, the system will eventually end up in one of the recurrent classes. Intuitively, the conditional entropy rate will then be dictated by the entropy rates which are realizable by any given recurrent class. If the recurrent classes are all trackable, then they will all have zero conditional entropy rate. If any recurrent class is not trackable, then there will be a nonzero conditional entropy rate.

To formalize this intuition, we start with the “if” part. Let the random variable correspond to which recurrent class is in. To ensure this is well-defined for all T, we take to also have a special value indicating that is a transient node. We can then define the stochastic process . Because is completely determined by , we have

We can then expand this as

Noting that we are interested in the limit as , will have a time where it transitions from transient nodes to one recurrent class, which we denote C. Therefore, any sufficiently long can be entirely defined by and C. We can then rewrite the conditional entropy as

This expression has a nice interpretation: the first term corresponds to the uncertainty in which recurrent class is entered and when it is entered, while the second term corresponds to the uncertainty in the nodes given the observed colors and the specific recurrent class and transition time.

Letting , we can write

We then have the following inequalities:

Taking the limit to obtain the conditional entropy rate, we obtain

The first term has only a polynomial number of possibilities for and a finite, constant number of possibilities for C. Therefore, the same reasoning used to prove Theorem 6 shows that this term will go to zero. The second term does not grow with T, and hence will be taken to zero by the 1/T in the limit. The third term can be expanded as

where and C are the sample spaces of and C, respectively. Once the limit is taken, each term in the sum corresponds to the conditional entropy rate produced by a given recurrent class c. Theorem 1 of [9] states that this entropy rate is zero for an irreducible, aperiodic, trackable model. Therefore, if all recurrent classes are trackable and all recurrent nodes are aperiodic, all terms in the sum are zero and H(X|Y ) = 0.

To prove the “only if” part, start with the inequality

For each term in the sum, we have

When we take the limit as , the first term only involves the uncertainty in the finite set of random variables , and hence will be taken to zero by the factor of 1/T. This means that the lower bound will be determined by , which is the entropy of the states in the recurrent part conditioned on all of the observations. Focusing on this term, we then have

Because of the conditional independence properties of hidden Markov models, we can drop the conditioning on : all of the information the symbols from the transient phase convey about is subsumed by . So,

But, is simply the entropy produced by transitions within recurrent class c with an initial state distribution dictated by the edges from x into c. Note that Geiger and Temmel assume the initial state distribution is equal to the stationary distribution, but they comment that their results should hold for any initial state distribution [9]. Indeed, the proof of Theorem 1 of [9] does not appear to depend on any specific choice of initial state distribution. Therefore, we can take the limit to obtain the conditional entropy rate and apply the results of Theorem 1 of [9] to each term in the sum to conclude that the only way for H(X|Y ) > 0 is for at least one of the recurrent classes to be untrackable.

This theorem implies that models which are untrackable because of pathologies in the transient nodes can still have H(X|Y ) = 0. This can be understood in terms of the difference between the worst-case and asymptotic average behavior: trackability considers whether there is any color sequence which can generate exponential growth in the size of the hypothesis set, whereas the conditional entropy rate captures the fact that, on asymptotic average, such sequences cannot persist infinitely long if they are only generated by transient nodes. To illustrate this, consider the model in Figure 9. This model has intersecting cycles with the same coloring formed by the transient nodes {a, b, c}, and therefore experiences exponential worst-case growth of the hypothesis set. But, the single recurrent node d has a unique color, so the hypothesis set stops growing as soon as is seen. Given our assumption that the probabilities assigned to the edges are strictly positive, this is guaranteed to happen eventually. At this point, there are no additional bits of uncertainty added at each time step.

Fig. 9: Untrackable model with H(X|Y ) = 0.

As a less obvious example, suppose the model in Figure 9 is modified such that all nodes emit the same color. Then it is never possible to identify when the recurrent node d has been entered, and yet Theorem 7 implies that H(X|Y ) = 0. In this case, the hypothesis set continues to grow exponentially, but the hypotheses which visit the transient nodes many times are increasingly less likely. Theorem 7 then indicates that the rate at which such hypotheses become unlikely is sufficient for the conditional entropy rate to be zero.

This paper has presented several new analytic results regarding the ability to reconstruct the sequence of nodes visited by an agent traversing a trackable weak model. It was found that the node reconstruction problem is substantially more tractable in models that are strongly connected compared to models that have transient nodes. This is a promising result because trackability imposes fewer constraints on the model’s coloring than the other observability classes which offer stronger performance guarantees [5], and hence is more likely to be attainable in practice. For strongly-connected trackable models, the worst-case size of the hypothesis set is bounded by an easily-computed constant. Furthermore, when unambiguous real-time tracking is lost because of a same-colored out-neighbor, it will always eventually be possible to determine which branch was taken. Finally it was shown that the conditional entropy rate, H(X|Y ), vanishes if and only if all recurrent classes are trackable. The implications of these theorems were elaborated in terms of well-known mean absorption and recurrence times of Markov chains that can be used to estimate how long certain events such as unique node identification can take on average.

These new results identify deep relationships between structural properties of weak models and lumped Markov chains, and their trackability and entropy rates. Consequently, the results of this paper can be used to concretely inform the design of trackable systems as well as the tracking algorithms used in applications such as computer program execution monitoring with out-of-band electromagnetic emissions measurements [3].

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

The research effort depicted was sponsored by the Air Force Research Laboratory (AFRL) and the Defense Advanced Research Projects Agency (DARPA) under the Leveraging the Analog Domain for Security (LADS) program under contract number FA8650-16-C-7622. In particular, we thank Dr. Angelos Keromytis and Mr. Ian Crone, the past and present DARPA program managers of LADS, for their encouragement and support throughout the program.

This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA).

The views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.

[1] L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proceedings of the IEEE, vol. 77, no. 2, pp. 257–286, 1989.

[2] V. Crespi, G. Cybenko, and G. Jiang, “The theory of trackability with applications to sensor networks,” ACM Transactions on Sensor Networks (TOSN), vol. 4, no. 3, p. 16, 2008.

[3] M. Chilenski, G. Cybenko, I. Dekine, P. Kumar, and G. Raz, “Control flow graph modifications for improved RF-based processor tracking performance,” in Cyber Sensing 2018, ser. Proc. SPIE, vol. 10630, 2018, p. 106300I.

[4] M. Kearns and L. Valiant, “Cryptographic limitations on learning Boolean formulae and finite automata,” Journal of the ACM (JACM), vol. 41, no. 1, pp. 67–95, 1994.

[5] M. Chilenski, G. Cybenko, I. Dekine, P. Kumar, and G. Raz, “Observability properties of colored graphs,” IEEE Transactions on Network Science and Engineering (Early Access), 2019.

[6] Y. Sheng and G. V. Cybenko, “Distance measures for nonparametric weak process models,” in Systems, Man and Cybernetics, 2005 IEEE International Conference on. IEEE, 2005.

[7] R. M. Jungers and V. D. Blondel, “Observable graphs,” Discrete Applied Mathematics, vol. 159, no. 10, pp. 981–989, 2011.

[8] L. Gurvits and J. Ledoux, “Markov property for a function of a Markov chain: A linear algebra approach,” Linear Algebra and Its Applications, vol. 404, pp. 85–117, 2005.

[9] B. C. Geiger and C. Temmel, “Lumpings of Markov chains, entropy rate preservation, and higher-order lumpability,” Journal of Applied Probability, vol. 51, no. 4, pp. 1114–1132, 2014.

[10] D. P. Bertsekas and J. N. Tsitsiklis, Introduction to Probability, 2nd ed. Athena Scientific, 2008, ch. 7.

[11] T. M. Cover and J. A. Thomas, Elements of Information Theory. John Wiley & Sons, 2012.

Mark Chilenski received the BS degree in aeronautical and astronautical engineering from the University of Washington in 2010 and the PhD degree in nuclear science and engineering from the Massachusetts Institute of Technology in 2016. He is a senior scientist at Systems & Technology Research LLC. His research interests include machine learning, Bayesian inference, and cybersecurity.

George Cybenko received his B.Sc. and Ph.D. degrees in Mathematics from the University of Toronto and Princeton. He is currently the Dorothy and Walter Gramm Professor of Engineering at Dartmouth. His research interests include cyber security, advanced machine learning algorithms and information deception.

Isaac Dekine received the BS and MS degrees in electrical and computer engineering from Carnegie Mellon University in 2006. He is a senior engineer at Systems & Technology Research LLC. His research interests include RF system design, signal processing, and cybersecurity.

Piyush Kumar received the Master of Science degree in Physics from the Indian Institute of Technology Kharagpur in 2001, MS degree in Physics from the University of Chicago in 2004 and the PhD degree in Physics from the University of Michigan Ann Arbor in 2007. He is a lead scientist at Systems & Technology Research LLC. His research interests include applications of probabilistic methods to problems in physics and engineering, machine learning, and graph theory.

Gil Raz received a bachelor’s degree in electrical engineering from the Technion – Israel Institute of Technology in 1988 and a PhD in electrical engineering (minor in mathematics) from the University of Wisconsin – Madison in 1998. He is chief scientist at Systems & Technology Research LLC. His research interests include applied mathematics and statistics for solving problems in multiple application areas.