Bayesian Tensor Network with Polynomial Complexity for Probabilistic Machine Learning

2019·arXiv

Abstract

Abstract

It is known that describing or calculating the conditional probabilities of multiple events is exponentially expensive. In this work, Bayesian tensor network (BTN) is proposed to efficiently capture the conditional probabilities of multiple sets of events with polynomial complexity. BTN is a directed acyclic graphical model that forms a subset of TN. To testify its validity for exponentially many events, BTN is implemented to the image recognition, where the classification is mapped to capturing the conditional probabilities in an exponentially large sample space. Competitive performance is achieved by the BTN with simple tree network structures. Analogous to the tensor network simulations of quantum systems, the validity of the simple-tree BTN implies an “area law” of fluctuations in image recognition problems.

I. INTRODUCTION

Tensor network (TN) is a powerful non-linear (more specifically, multi-linear) model [1] that was originally developed in the field of quantum many-body physics [2–4]. With the striking resemblances to neural network (NN), TN has recently been applied to machine learning [5–10]. Treated as quantum many-body states or operators [11, 12], TN is expected to provide a natural solution for probabilistic machine learning [13]. Therefore, TN is also called Born machine [14], as the quantum counterpart of the famous Boltzmann machine.

Aside from the probabilistic interpretation, one significant advantage of TN is its efficiency. For quantum many-body simulations, the complexity of a many-body state and the dimension of the Hilbert space scale exponentially with the system size. TN “magically” reduces the complexity to only scale polynomially [15]. The validity of TN is based on the area law of the entanglement entropy, where the important correlations or fluctuations are short-range [16, 17]. Then TN uses only polynomially expensive resources to accurately capture the short-range fluctuations and obtains high accuracies.

This work concerns the conditional probabilities of multiple event, for which we have another well-established probabilistic model known as Bayesian belief network (BBN) [18]. Classifications and inferences can be done with BBN by implementing Gibbs sampling after optimizing the network structure from the data. In this way, the causal relations can be inferred from the conditional probabilities among the nodes. However, either calculating the conditional probabilities of multiple events or obtaining the global optimal structure of the BBN is exponentially expensive or NP-hard [19, 20]. One solution is to combine BBN with the ideas of deep learning, which has a long history [21, 22] and is currently under hot debate (see, e.g., [23, 24]).

In this work, Bayesian tensor network (BTN) is proposed for probabilistic machine learning. BTN is a directed acyclic graphical model that forms a subset of TN (see a simple-tree BTN in Fig. 1 (a) as an example); it inherits the advantages of both BBN and TN. BTN provides an efficient representation for the conditional probabilities of massive events with

FIG. 1. (Color online) (a) The illustration of image recognition by a simple-tree BTN of two layers. A () image is mapped to (sets of events as the root sets. The first layer contains four tensors that map the root sets (blue) to () hidden sets (yellow). The last layer contains one tensor that maps the hidden sets to the leaf set (red) that indicates the probability distribution of the classification of the image. (b) By using the tail-to-tail structure, the first layer can be designed to contain () tensors, which map the root sets to () hidden sets.

only polynomial complexity [see, for instance, Eqs. (15) and (16)]. The structure of a BTN is priorly designed, and the performance does not severely depend on the structure. The casual relations among the events can be inferred by calculating the conditional probabilities using tensor contractions instead of Gibbs samplings. A rotation optimization method [25] is suggested to update BTN, which avoids gradient vanishing problem and exhibits high efficiency. To testify BTN, the image recognition is mapped to a probabilistic problem, i.e., to capture the conditional probabilities in an exponentially large sample space. BTN exhibits competitive performances on the fashion-MNIST dataset [26]. The expressive power of BTN is discussed from the perspectives of the probabilities of events, the dimension of the vector space where the BTN is defined, and the “area law” [15, 17, 27–30] of the fluctuations of TN.

FIG. 2. (Color online) Illustrations of (a) a Bayesian tensor (b) the contraction of two Bayesian tensors (c) head-to-head structure, and (d) tail-to-tail structure.

II. BAYESIAN TENSOR NETWORK: DEFINITIONS AND ALGEBRAS

Before define Bayesian TN, let me firstly define Bayesian tensor. Considering a set (denoted as X) of mutually exclusive events (denoted as ), a vector V is used to describes the probabilities of these events as the following. V is positively defined, and its i-th element gives the probability of the i-th event as . We have

A Bayesian tensor T (see several related works by tensor decompositions at [31–34]) describes the conditional probabilities of multiple sets of mutually exclusive events. Here, let us consider ( ) sets of events (m = ) and . We have

where T is a ( )-th ordered tensor and are its indexes. The dimension of each index equals to the number of events in the corresponding set, i.e., . A Bayesian tensor has a direction from to . In Fig. 2, we use squares to represent a Bayesian tensor, and use the bonds connecting with a square to represent the indexes. On each bond, we put a solid triangle to represent the set of events and to indicate the direction. The indexes are dubbed as in-going indexes; j is dubbed as out-going index.

satisfy the following properties: (a) positivity: T is positively defined. This is a direct result from Eq. (2); (b) normalization: for any in-going indexes (), it has

where “:” means to go through all values of the corresponding index. Eq. (3) means .

A tensor satisfying (b) must have the following property: (b) for any vectors with and m = , we have with

with . Then means . Note it is assumed here that each two events from different sets (corresponding to the in-going indexes) are independent, i.e., when and with . If two sets contain nonindependent events, one needs to replace the products by the joint probability distributions.

(c) Direction inversion. Knowing the probability distribution of the sets, the directions of the indexes of a Bayesian tensor can be inverted by using Bayes’ equation. For instance, consider three sets of events (, and ) and the Bayesian tensor giving the conditional probabilities as . Assume two probability distribu- tions and are priorly known (and are independent to each other). We have the total joint probability distribution . Then we have the probability distribution of as [also see Eq. (4)], and the joint distribution . From Bayes’ equation, the direction inversion is done as

The equations above are the tensor versions of Bayes’ equation.

The definition of Bayesian tensor network (BTN) is very simple: the contractions of multiple Bayesian tensors. BTN is a well-defined subset of TN. This is different from the generalized TN’s such as string-bond states which contain non-multi-linear operations [11, 35, 36]. This is the reason why BTN does not require samplings.

BTN can be deep. Graphically, we use a shared bond with a triangle on it to represent the same event in different Bayesian tensors. To proceed, let me introduce the concepts of root sets, leaf sets, and hidden sets. Following the directions of the triangles, the root and leaf sets are the events at the starting and ending points of the whole BTN, respectively. The rest are the hidden sets. For simplicity, we assume that the network graph of a BTN is connected. In the following, let me show several important properties of BTN.

(a) Connection rules. Consider that a set appears at more than one Bayesian tensors. If this set corresponds to an in-going index for all the connected tensors, one should use the tail-to-tail structure to connect to the tensors; if this set corresponds to an out-going index for all the connected tensors, one should use the head-to-head structure to connect.

(b) Contraction rule. If a set appears at two Bayesian tensors as in-going and out-going index, respectively, one should connect these two tensors by a bond; this index represents a hidden set and should be contracted in the TN computations. Note a hidden index is analogous to a virtual or ancillary bond in the TN terminology [2].

(c) Directed acyclicality. BTN is a directed acyclic graphical model. This property is the same as the BBN. A circle in a BTN or BBN is forbidden, meaning if one goes along the directions of the triangles, one cannot go back to the tensors that have been got through. If a circle appears in the BTN, the causal relations will become inconsistent. Note that loops are defined as the loops of the network graph, same to the definition in the TN’s. There are no constraints concerning the loops as long as the model is directed acyclic. Different structure with loops can be designed applying the connection and contraction rules.

(d) Contraction consistency. Given a BTN formed by two Bayesian tensors and , the tensor by contracting the shared index as is a Bayesian tensor. Fig. 2 (b) il- lustrates an example. This can be readily proven using the probabilistic language of the Bayesian tensors. An inference can be made from the contraction consistency: (d) A BTN can be written as one Bayesian tensor. This inference can be proven by contracting all the shared indexes (hidden sets) in the BTN. An equivalent description of (d) is that a BTN represents the conditional probabilities between the leaf and root sets.

(e) Relation between loop and independence. Assume that different root sets are independent. If different in-going sets of one Bayesian tensor in the BTN are dependent, there must exist at least one loop between this tensor and the root sets. An equivalent expression is that if the BTN is loop-free, any different in-going sets in one tensor are independent.

(f) Probability distributions of hidden and leaf sets. Assume the probability distributions of the root sets are known, and there exist no loops in the BTN. Given a BTN, the probability distributions of all hidden and leaf set can be calculated. This can be done by contracting the BTN from the roots to the leafs following the directions of the triangles. If loops appear, one will get joint probability distributions of multiple sets, instead of the probability distribution of each single set.

(g) Inferences. When property (f) holds, the conditional probabilities among any two root sets can be calculated. This can be done by using Bayes’ equation (see Eqs. (5) and (6) as examples) to invert the Bayesian tensors on the path connecting the two root sets. Note that for BBN, the network structure is optimized according to the data. The network structure of BTN is priorly determined. We only optimize the parameters in the Bayesian tensors, not the structure. For inferences, the conditional probabilities among the root sets can be calculated after the Bayesian tensors are optimized. Meanwhile, samplings are not needed for BTN. Instead, one should implement tensor contractions.

(h) Parameter complexity. The parameter complexity of a BTN is defined as the total number of parameters of the tensors. The parameter complexity scales polynomially with the total number of sets and (approximately) the average number of events in one sets (i.e., the dimensions of the indexes). For different network structures, the parameter complexity will be slightly different. In comparison, the parameter complexity of the full conditional probabilities scales exponentially with the number of sets.

(i) Contraction complexity. The contraction complexity of a BTN is to characterize the complexity brought by the loops of the network graph. Without performing any approximations (such as those in the TN contraction algorithms; see, e.g., [37, 38]), the complexity increases exponentially with the number of loops. This property is the same as the standard TN contractions. In addition, the loops also increase the cost to calculate and save the chains of gradients in the back propagation (if implemented).

III. PROBABILISTIC IMAGE RECOGNITION AND ROTATION OPTIMIZATION ALGORITHM

To testify BTN, let me use it for supervised learning by taking image recognition as an example. The central task is to capture the conditional probabilities between the images and their classifications. The dimension of the sample space scales exponentially with the number of features.

The first step is to map each feature (pixel; let me denote the m-th pixel in the n-th image as ) to a set of exclusive events . The probability distribution of these events is determined by the value of the pixel. One simplest example is the binary images, where the events are taking the pixel to be 0 (black) or 1 (white). Obviously a pixel cannot be black and white at the same time, thus these events should be mutually exclusive. For a given pixel , we can set the probability distribution as and . Thus the vector to put at the root is two-dimensional that satisfies

The map from features to the probability distributions of sets of events, e.g., Eq. (7), is dubbed as the probabilistic map.

For the gray-scale images with , one choice is to use the following map, which has been used in TN machine learning [5], to define d-dimensional ’s as

It meas that one can control the number (i.e., d) of the mutually exclusive events in a set. With Eq. (8), one has . Therefore, we have , so that and .

With the probability distributions of the root sets using the probabilistic map, the next step is to use BTN to capture the conditional probabilities between the root and leaf sets. One simplest example of a tree BTN that has been used in TN mahine learning [6] is illustrated in Fig. 1 (a). For classifica-tion, there is only one leaf set, where the number of the events equal to the number of classes. For a tree, the network graph has no loops. If any hidden set of events are known (fixed), the root sets will be blocked into two independent parts. Note loops can be designed by adding tail-to-tail structures. Fig. 1 (b) shows an example, where the () root sets are mapped to () hidden sets after the first layer. This, however, will inevitably increase the contraction complexity (see property (i) of BTN).

The loss function for classification can be chosen as the cross entropy, same as the NN’s or TN’s. For a NN, one usually implements softmax on the outputs to satisfy the normalization of the probability. For a BTN, this is not necessary since the normalization is naturally fulfilled (see property (d) or (d) of BTN).

The tensors in the BTN can be updated by the gradient methods, i.e., with f the loss function and the learning rate. However, our data imply that the methods to control the gradients of NN’s (e.g., Adam [39]) works badly for BTN. This is possibly due to the normalization constraints given in Eq. (3).

In the following, a gradient optimization method is proposed by taking advantage of the normalization constraints, which is dubbed as rotation optimization. The central idea is to rotate the normalized vectors, where the learning rate is controlled by the rotation angel [25]. To this end, I introduce the ancillary tensor

As T is positively defined, Q should be real. We have . The rotation optimization is to update Q instead of T.

Firstly, one calculates the gradient of one ancillary tensor (say ) by back propagation as . Note G is also a tensor (not a Bayesian tensor) with the same order as Q and T. Then calculate the components that are orthogonal to the normalized vectors in Q as

We have the orthogonality between G and Q as

It means that with fixed in-going indexes (), the vector is orthogonal to . Then normalize G as

This normalization does not change the orthogonality between G and Q. Then update Q by G as

where is the rotation angel according to the triangular relation between G and Q. Finally, normalized Q as

so that T satisfies Eq. (3). The potential gradient vanishing problem are avoided by the rotation optimization, since the changes of the normalized vectors are strictly controlled by the rotation angel.

IV. DISCUSSIONS AND RESULTS: EXPRESSIVE POWER AND AREA LAW

I choose fashion-MNIST dataset to testify [26]. These images are gray-scale with the definition , and belong to ten categories. We have the number of root sets and the dimension of the leaf set . The network structure is chosen as a tree similar to Fig. 1 (a) (also see [6]). For simplicity, assume that the dimension of every root index (i.e., the number of events in every root set) is the same, denoted by d; the dimension of every hidden index equals to . Then the dimension of the sample space equals to . Obviously, it is extremely challenging for BBN to handle such a problem. For the BTN, each Bayesian tensor is chosen to have four in-going indexes and one out-going index. The parameter complexity of such a BTN scales as

Another network structure is also tried, where each tensor has two in-going indexes and one out-going index (one may find a similar tree structure in [10]). The complexity satisfies

The complexities of both BTNs’s scale polynomially with the number of sets of events (M) and the numbers of events in each set (d and ). Note one can access larger for the second BTN as it obviously possesses less parameter complexity. However, the number of layers in the second BTN is doubled, thus possesses higher contraction complexity (e.g., deeper back propagation of the gradients).

The expressive power of a BTN should be limited by the dimensions of the root sets d, the dimensions of the hidden sets , and the network structure. Let us firstly discuss the upper bound of the expressive power. If the number of the events in a root set d equals to the number of values that a feature can take, the expressive power reaches the genuine upper bound. In this case, all events are represented as orthonormal basis of a vector space, meaning all events are mutually exclusive. There are in total mutually exclusive events, corresponding to orthonormal basis of the vector space.

However, d is usually much smaller than the number of values that a feature can take. For instance, one may take d = 4 for gray-scale images using Eq. (7) or (8), in which a pixel can take 256 values. In this case, the events that should be mutually exclusive become non-exclusive. For example, one pixel cannot simultaneously be black and gray in a given image, but the events of the pixel being black or gray are not exclusive for d < 256. Mathematically, the vectors V [Eq.

FIG. 3. (Color online) The training and testing accuracies against the root bond dimension d, while taking the hidden bond dimension . About 1000 epochs are taken in each simulation.

(7) or (8)] for the pixel being black or gray are not orthogonal to each other. In this case, the upper bound of the expressive power of the BTN is determined by d.

The dimension of the hidden indexes determines how well the upper bound is approached. To see this, let us consider the Bayesian tensor T that corresponds to the whole BTN (see the property (d) of the BTN). The total dimension of its in-going indexes equals to . Such a tensor can be considered as normalized vectors defined in the vector space whose dimension is .

If one is able to properly process the full T , in principle the upper bound can be reached. However, T is exponentially large. The BTN then can be regarded as an efficient approximation of the full T by writing it in a specific TN form. In other words, the BTN can be consider as a tensor cluster decomposed from T [31–34], if such a decomposition exists. The dimension of the hidden indexes controls how accurately the BTN can approximate the T at the upper bound. The same argument about the expressive power on the TN machine learning has been given in Ref. [6].

As discussed above, any connected network structure can approach the upper bound with sufficiently large . This might be the reason that the expressive power of BTN does not severely depend on the network structure, unlike the NN’s. For a BTN, the network structure should affect how fast the upper bound can be approached by increasing . For simplicity, let us consider the loop-free structures. As we know from the TN, the fluctuations between two sets of events (formally written as with the average of x; same to the covariance) decays exponentially with the length of the path connecting these two indexes. This is a fundamental property of a loop-free TN. With a good structure, the sets with large covariance should be connected by a shorter path. Then it would require a smaller to capture the fluctuations among the root and leaf events.

Based on the above arguments, a conjecture about the bottleneck of the expressive power is proposed as the following: the bottleneck of a given network structure (namely structural bottleneck) is reached when the performance of the model in-

FIG. 4. (Color online) Training and testing accuracies versus the hidden dimension . The root dimension is taken as d = 2 and 4.

creases exponentially slowly with the parameter complexity (see property (h) of BTN). Fig. 3 shows the training and testing accuracies against d (taking ). Both accuracies have not converged for the largest d that our at-hand hardware can process. It means that the structural bottlenecks of such simple tree networks have not been reached yet.

Fig. 4 shows the accuracies versus the hidden dimension with fixed root dimension (d = 2 and 4). The accuracies increase with , meaning the expressive power of the BTN is approaching the upper bound of the given d. One can see that even for d = 2, it is not easy to reach the upper bound, as the dimension of the vector space scales as .

In general, BTNoutperforms BTNwhile taking the same d and . There are mainly two reasons. One is that the parameter complexity of BTNis higher than BTNwhen d and are the same. The other is that the lengths of the paths connecting two root indexes in BTNare mostly doubled in BTN, since the depth of the tree is doubled. As the fluctu-ations decay exponentially with the lengths, it would require larger for BTNto capture the fluctuations.

As the dimension of the vector (or sample) space is exponentially large, BBN cannot be implemented on such problems efficiently. One choice is a simplified Bayesian model called Naive Bayesian classifiers by assuming the independence of the events. BTN significantly surpasses Naive Bayesian classifiers with the testing accuracy around 70% [40]. This implies that the dependences among the events are important and well-considered by BTN. In addition, it is widely accepted that probabilistic models (including BBN) are more suitable for inferences than classifications. Thus it is no surprise that BTN does not beat the state-of-the-art NN on image recognition. Note for BTN even for image recognition, there are many ways to further improve the accuracy by, e.g., pushing to larger numbers of parameters with more computational resources and/or designing more complicated and deeper network structures.

“Area law” means a quantity scales only with the boundary, not the bulk. The area law of entropy or entanglement appears in many systems, such as black holes [27] and quantum states [17]. For quantum many-body simulations, a TN, including

FIG. 5. (Color online) The training and testing accuracies for differ- ent epochs with BTN. For the rotation optimization, a fair convergence is reached after about 100 epochs. For Adam, the parameters seem to be trapped to a bad local minimum. Both learning rates are taken as

the tree TN, naturally satisfies the area law of entanglement entropy. The validity of the simple tree BTN implies an area law of covariance in the image recognition problems. If one separates the image into two parts by drawing a boundary, the main contributions of the covariance between these two parts should be from the short-range covariances of the data near the boundary. The BTN accurately captures these covariances and gives accurate results of classification. On the other hand, the performance of BTN can be improved by considering the covariances in larger ranges. Possible solutions include modifying the network structure, and pre-processing the features to localize the covariances.

Finally, Fig. 5 demonstrates the convergences of the accuracies by the rotation optimization and Adam. For Adam, we use the standard procedure to update the tensors, and then normalize the tensors by hand to satisfy Eq. (3). Adam seems

to suffer the gradient vanishing problem and fails to give reasonable results. Note that I am not stating that Adam cannot work to optimize BTN. It is possible to obtain a good performance by carefully designing the optimization process based on Adam (e.g., by adding Lagrangian terms). From the data at hand, the rotation optimization method is more robust and efficient.

V. SUMMARY

This work aims at proposing Bayesian tensor network (BTN) for probabilistic machine learning. For capturing the conditional probabilities among multiple sets of events, BTN exhibits high efficiency and accuracy with polynomial complexity when the sample space is exponentially large. An optimization method is proposed to efficiently update BTN, which avoids the potential gradient vanishing problem. Our results imply an area law of the covariance in the image recognition problems. As an efficient model, I expect that BTN will be useful not only for machine learning but also for the physical simulations by TN.

ACKNOWLEDGMENT

SJR is grateful to Ding Liu, Zheng-Zhi Sun, and Ye-Ming Meng for helpful discussions. This work was supported by Beijing Natural Science Foundation (Grant No. 1192005 and No. Z180013) and the NSFC (Grant No. 11834014), and by the Academy for Multidisciplinary Studies, Capital Normal University.

The code of BTN is publicly available on GitHub: https: //github.com/ranshiju/BayesianTN.

[1] S.-J. Ran, E. Tirrito, C. Peng, X. Chen, L. Tagliacozzo, G. Su, and M. Lewenstein, Tensor Network Contractions (to be published on Springer - Lecture Notes in Physics, Heidelberg, 2019) arXiv:1708.09213.

[2] F. Verstraete, V. Murg, and J. I. Cirac, Matrix product states, projected entangled pair states, and variational renormalization group methods for quantum spin systems, Advances in Physics 57, 143 (2008).

[3] J. I. Cirac and F. Verstraete, Renormalization and tensor product states in spin chains and lattices, J. Phys. A: Math. Theor. 42, 504004 (2009).

[4] R. Or´us, Tensor networks for complex quantum systems, Na- ture Reviews Physics 1, 538 (2019).

[5] E. Stoudenmire and D. J. Schwab, Supervised learning with ten- sor networks, in Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Curran Associates, Inc., 2016) pp.

4799–4807.

[6] D. Liu, S.-J. Ran, P. Wittek, C. Peng, R. B. Garc´ıa, G. Su, and M. Lewenstein, Machine learning by unitary tensor network of hierarchical tree structure, New Journal of Physics 21, 073059 (2019).

[7] Z.-Y. Han, J. Wang, H. Fan, L. Wang, and P. Zhang, Unsupervised generative modeling using matrix product states, Phys. Rev. X 8, 031012 (2018).

[8] C. Guo, Z. Jie, W. Lu, and D. Poletti, Matrix product operators for sequence-to-sequence learning, Phys. Rev. E 98, 042114 (2018).

[9] E. M. Stoudenmire, Learning relevant features of data with multi-scale tensor networks, Quantum Science and Technology 3, 034003 (2018).

[10] S. Cheng, L. Wang, T. Xiang, and P. Zhang, Tree tensor networks for generative modeling, Phys. Rev. B 99, 155131 (2019).

[11] I. Glasser, N. Pancotti, and J. I. Cirac, Supervised learning with generalized tensor networks, (2018), arXiv:1806.05964.

[12] I. Glasser, R. Sweke, N. Pancotti, J. Eisert, and I. Cirac, Ex- pressive power of tensor-network factorizations for probabilistic modeling, in Advances in Neural Information Processing Systems 32 (Curran Associates, Inc., 2019) pp. 1496–1508.

[13] Z. Ghahramani, Probabilistic machine learning and artificial in- telligence, Nature 521, 452 (2015).

[14] S. Cheng, J. Chen, and L. Wang, Information perspective to probabilistic modeling: Boltzmann machines versus born machines, Entropy 20, 583 (2018).

[15] R. Or´us, A practical introduction to tensor networks: Matrix product states and projected entangled pair states, Ann. Phys. 349, 117 (2014).

[16] I. Bengtsson and K. ˙Zyczkowski, Geometry of quantum states: an introduction to quantum entanglement (Cambridge university press, 2017).

[17] J. Eisert, M. Cramer, and M. B. Plenio, Colloquium: Area laws for the entanglement entropy, Rev. Mod. Phys. 82, 277 (2010).

[18] N. Friedman, D. Geiger, and M. Goldszmidt, Bayesian network classifiers, Machine learning 29, 131 (1997).

[19] P. Dagum and M. Luby, Approximating probabilistic inference in bayesian belief networks is np-hard, Artificial Intelligence 60, 141 (1993).

[20] D. M. Chickering, Learning bayesian networks is np-complete, in Learning from data (Springer, 1996) pp. 121–130.

[21] D. Barber and C. M. Bishop, Ensemble learning in bayesian neural networks, Nato ASI Series F Computer and Systems Sci- ences 168, 215 (1998).

[22] R. M. Neal, Bayesian learning for neural networks, Vol. 118 (Springer Science & Business Media, 2012).

[23] K. Osawa, S. Swaroop, M. E. E. Khan, A. Jain, R. Eschen- hagen, R. E. Turner, and R. Yokota, Practical deep learning with bayesian principles, in Advances in Neural Information Pro- cessing Systems 32 (Curran Associates, Inc., 2019) pp. 4289– 4301.

[24] A. Sobolev and D. P. Vetrov, Importance weighted hierarchical variational inference, in Advances in Neural Information Pro- cessing Systems 32 (Curran Associates, Inc., 2019) pp. 601– 613.

[25] Z.-Z. Sun, S.-J. Ran, and G. Su, Tangent-space gradient opti- mization of tensor network for machine learning, in preperation (2019).

[26] H. Xiao, K. Rasul, and R. Vollgraf, Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, (2017), arXiv:1708.07747.

[27] J. D. Bekenstein, Black Holes and Entropy, Phys. Rev. D 7, 2333 (1973).

[28] M. Srednicki, Entropy and area, Phys. Rev. Lett. 71, 666 (1993).

[29] M. B. Plenio, J. Eisert, J. Dreissig, and M. Cramer, Entropy, Entanglement, and Area: Analytical Results for Harmonic Lattice Systems, Phys. Rev. Lett. 94, 060503 (2005).

[30] A. J. Ferris, Area law and real-space renormalization, Phys. Rev. B 87, 125139 (2013).

[31] D. Tao, M. Song, X. Li, J. Shen, J. Sun, X. Wu, C. Faloutsos, and S. J. Maybank, Bayesian tensor approach for 3-d face modeling, IEEE Transactions on Circuits and Systems for Video Technology 18, 1397 (2008).

[32] I. Sutskever, J. B. Tenenbaum, and R. R. Salakhutdinov, Modelling relational data using bayesian clustered tensor factorization, in Advances in Neural Information Processing Systems 22, edited by Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (Curran Associates, Inc., 2009) pp. 1821–1828.

[33] L. Xiong, X. Chen, T.-K. Huang, J. Schneider, and J. G. Carbonell, Temporal collaborative filtering with bayesian probabilistic tensor factorization, in Proceedings of the 2010 SIAM international conference on data mining (SIAM, 2010) pp. 211–222.

[34] Q. Zhao, L. Zhang, and A. Cichocki, Bayesian cp factorization of incomplete tensors with automatic rank determination, IEEE transactions on pattern analysis and machine intelligence 37, 1751 (2015).

[35] N. Schuch, M. M. Wolf, F. Verstraete, and J. I. Cirac, Simu- lation of quantum many-body systems with strings of operators and monte carlo tensor contractions, Phys. Rev. Lett. 100, 040501 (2008).

[36] I. Glasser, N. Pancotti, M. August, I. D. Rodriguez, and J. I. Cirac, Neural-network quantum states, string-bond states, and chiral topological states, Phys. Rev. X 8, 011006 (2018).

[37] M. Levin and C. P. Nave, Tensor renormalization group ap- proach to two-dimensional classical lattice models, Phys. Rev. Lett. 99, 120601 (2007).

[38] R. Or´us and G. Vidal, Simulation of two-dimensional quantum systems on an infinite lattice revisited: Corner transfer matrix for tensor contraction, Phys. Rev. B 80, 094403 (2009).

[39] D. P. Kingma and J. Ba, Adam: A method for stochastic opti- mization, in 3rd International Conference on Learning Repre- sentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015).

[40] M. J. Islam, Q. M. J. Wu, M. Ahmadi, and M. A. Sid-Ahmed, Investigating the performance of naive- bayes classifiers and knearest neighbor classifiers, in 2007 International Conference on Convergence Information Technology (ICCIT 2007) (2007) pp. 1541–1546.

Designed for Accessibility and to further Open Science