The study of events has long been a focus of many disciplines, including philosophy, cognitive psychology, linguistics, computer science, and AI. (Tulving 1983) postulated a separate cognitive process for event recognition called episodic memory. In natural language, events have been studied from many different approaches, from formal logic and AI (Allen 1984), to computational linguistics (Pustejovsky et al. 2011). In computer science, event representations has been acquired by means of classification (Shahroudy et al. 2016) or represented as the composition of primitive actions (Veeraraghavan, Papanikolopoulos, and Schrater 2007).

In this paper, our focus lies within a smaller and more restricted set of human activities involving human and object interactions. We use a fine-grained event capture and annotation framework (Do and Pustejovsky 2017) as our basic setup for event classification. This framework takes into account both subeventual structures and extra-verbal factors (Pustejovsky 1995) in treating the classification of events (considering the difference between actions such as ”jump on” and ”jump over”). This allows us to investigate the effects of QSR on our event classification framework.

In particular, we use an event capture and annotation tool called ECAT (Do, Krishnaswamy, and Pustejovsky 2016), which employs Microsoft Kinectto capture sessions of performers interacting with two types of objects in our Block World environment, a cube (which can be slid on a flat surface) and a cylinder (which can be rolled). Objects are tracked using markers fixed to their sides facing the camera. They are then projected into three dimensional space using Depth of Field (DoF). Performers are tracked using the KinectAPI, which provides three dimensional inputs of their joint points (e.g., wrist, palm, shoulder). Event sessions are first sliced, and each slice is then annotated with a textual description with our event language. The event descriptions are in turn parsed into tuples of semantic roles.

We then transform the raw data into a spectrum of feature types. The first type consists of quantitative features reflecting positions of humans and objects projected on 3-dimensional and 2-dimensional feature spaces. The second type is built on top of the first, producing qualitative spatial (QS) representations for each image frame. The third type is a QS representation for each whole event duration, summarized on top of second-type features. Subsequently, we then compare our ML methods on 7 different kinds of features.

Depending on the form of extracted features (sequence vs. non-sequence), two different machine learning methods are implemented. For frame-sequence learning, we use Longshort term memory (LSTM); for whole-event learning, we used Multilayer perceptron model (MLP). Their similar neural network structures allow us to compare the two methods in a fair manner. In addition, we add a layer of constraints using a Conditional Random Field (CRF) algorithm before generating outputs.

The main contributions of our study are twofold. First, we propose a framework for event recording, annotation and classification, that achieves high accuracy using qualitative spatial reasoning for feature extraction. Second, by analyzing different levels of feature representations, from dense and continuous to sequential and discretized to summary event-level features, we determine the most economical and effective way for classification of human-object interaction.

Human activities such as running, sitting, eating, and playing sport have been investigated in previous research, such as (Shahroudy et al. 2016) and (Dubba et al. 2015). These human activities have significantly different motion signatures and durations. In some cases, algorithms learning to distinguish these events are actually learning the distinction between background or color histograms. Recently, some studies have begun to introduce datasets with more complex activities, especially involving human-object interactions, such as cooking activities (Rohrbach et al. 2012), or human-human interactions, such as hand shaking (Ryoo and Aggarwal 2010). More recently, (Li and Fritz 2016) investigates the possibility of predicting partial activities using a hierarchy label space. These studies have gradually led to a more fine-grained treatment of event classification.

To facilitate event classification, it is necessary to present events in a learnable format. This introduces the question of how to represent events, namely the difficulty in defining their temporal and spatial extensions, as well as the difficulty in selecting an observational perspective. For example, it is generally hard to demarcate an event duration from other events building up to it or its consequences. He kicks the ball may or may not involve the person running up to the ball, or include the ball flying to the goal. Similarly, to pick out a set of objects to include in an event representation is not trivial, even for events described in a text. For example, in He fried an egg, should we include the frying pan in the event representation or not? Point of view (POV) is also typically under-specified, even though there can be an infi-nite number of ways to interpret an event, depending on the rendering location. This leads to many different approaches in the representation of events for classification in computer science and machine learning. Events can be represented atomically, i.e., entire events are predicted in a classifica-tion manner (Shahroudy et al. 2016), or as combinations of more primitive actions (Veeraraghavan, Papanikolopoulos, and Schrater 2007), i.e., complex event types are learned based on recognition of combined primitive actions. For the former type of event representation, there are quantitative approaches based on low-level pixel features such as in (Le et al. 2011) and qualitative approaches such as induction from relational states among event participants (Dubba et al. 2015). For the latter approach, systems such as (Hoogs and Perera 2008), use state transition graphical models such as Dynamic Bayesian Networks (DBN).

Event classification using qualitative spatial methods has been discussed a fair amount in the literature. (Suchan, Bhatt, and Santos 2013) use the Regional Connected Calculus (RCC5) adjusted for depth field with data also recorded by Kinectsensor. This work classifies events related to people moving, sitting, or passing each other. (Dubba et al. 2015) provide an interesting framework which gives a summary explanation for a sequence of observations by alternating between inductive and abductive commonsense reasoning. This work was applied for two activity types, one being activities happening at an airport as boundary boxes of aircraft and trucks being tracked, and one of humans in-

teracting with an object and with each other.

For our study, we use the following feature types from QSRLib:

• CARDINAL DIRECTION, (Andrew, Mark, and White 1991) (QSRLIb cardir) measures compass relations between two objects into canonical directions such as North, North East etc.

• MOVING or STATIC (QSRLIb mos) measures whether a point is moving or not.

• QUALITATIVE DISTANCE CALCULUS (QSRLib argd) discretizes the distance between two moving points. There is a significant literature supporting the use of discretization for feature embedding. (Yang and Webb 2009) shows that “discretization is equivalent to using the true probability density function”. More recently, (Jiang et al. 2017) has used this method for classification of GPS trajectory. They studied three different approaches for discretization, including equal-width binning, Recursive Minimal Entropy Partitioning (RMEP) (Dougherty et al. 1995) and fuzzy discretization(Roy and Pal 2003). Their finding is that the equal-width binning approach is both simple and effective, so we used this approach with an interval length of 1/20 meter in embedded space. We did not explore other interval lengths, leaving that for future experiments.

• QUALITATIVE TRAJECTORY CALCULUS (Double Cross) (QSRLib qtccs): is a representation of motions between two objects by considering them as two moving point objects (MPOs) (Delafontaine, Cohn, and Van de Weghe 2011). The type C21 of (implemented in QSRLib) considers whether two points are moving toward each other or whether they are moving to the left or to the right of each other. Apparently, this is the kind of spatial semantics needed to learn the prepositions we used in this experiment. The following diagram explains this:

produces a tuple of 4 slots (A, B, C, D), where each could be given either -, + or 0, depending on the angle . For example, C is + if , if and 0 otherwise. QSRLib also allows specification of a quantisation factor , which dictates whether the movement of a point is significant in comparison to the distance between k and l.

Modification of Qualitative trajectory calculus Double Cross () implementation in QSRLib:

is not the most appropriate calculus for use in event classification without some modification, since ap-

Figure 1: Downstream feature extraction methods used in this study. Our focus is on the performance gain from quantitative features to qualitative features through the use of QSRLib and its extension for 3-dimensional data

Figure 2: 3D Grid around a point. N, S, W, E, B, A stand for North, South, West, East, Below, Above.

proximating objects as MPOs leads to loss of representational information. Exact modeling of object 3D volumes are feasible but cumbersome.

Our solution for this involves an angle quantisation factor which should be relatively small. When or , we set the the value of C to 0. Similarly for slot A, when or , we also set the value to be 0.

3-dimensional extension We extended QSRLib for the following feature types:

• CARDINAL DIRECTION (3D): We followed the 3D grid approach to partition the space into 3x3x3=27 voxels, (Sabharwal and Leopold 2014). The center voxel is the Minimum Bounding Hyper-rectangle (MBHR) of a reference object. In this framework, given two objects A (reference object) and B (target object), the cardinal direction from A to B is calculated firstly by generating the MBHR of A (it can be approximated), then setting up the 3D grid for A, then finding which voxels intersect with B. We actually used a much simpler alternative that replaces B with its centroid, so that only one direction is resulted.

Figure 3: Frenet-Serret frame for 3-D qualitative trajectory calculus

• QUALITATIVE DISTANCE CALCULUS (3D): This is basically the same as for two dimensional.

• QUALITATIVE TRAJECTORY CALCULUS (3D): It is noted that there are no features analogous to lateral slots (C, D) for 3-dimension. (Mavridis et al. 2015) provides an alternative for lateral relations, using Frenet-Serret frames (FS frame). We do not give the details of the calculation here, but rather provide our implementation. In a nutshell, the calculation steps are as follows:

– For each point P and Q, calculate the tangent vector, the binormal vector and the normal vector. For continuous data domain, this requires calculating second derivative of each point’s moving curve. For discrete domain, this translates to taking three data points into account for each calculation, of the current and two previous time steps. These three vectors create a FS frame for each moving point.

– Assuming that two FS frames and are calculated (special values are assigned in the case of degeneration), we need to find a transformation matrix from to . This rotation matrix is in turn decomposed into three values of yaw, pitch and roll angles. These three values, together with two feature values (A, B)

Figure 1 shows our downstream feature extraction methods.

Our motivation for creating downstream features is of the following basic intuition. • Object Model: State-by-state characterization of an object

as it changes or moves through time. • Action Model: State-by-state characterization of an ac-

tors motion through time. When action involves multiple

objects, this also includes effect of objects on each other. • Event Model: Composition of the object model with the

action model.

From a recognition point of view, the object model is translated to inherent motion of objects whereas the action model is translated to inter-object relative motion.

Here, R stands for human body rig, and stands for objects. This method factorizes the model representation into terms where n is the number of objects. This is not a very economical representation when there is a large number of objects in the scene (), but in reality, the number of objects is relatively static to each other, or we in fact need to consider a smaller number of objects to allow for possible descriptions (think of “Lunar eclipse occurs when the Moon passes directly behind the Earth into Earth’s shadow, aligning with the sun and Earth”, where we in fact do not take into account movement of the other planets).

Preprocessing The raw data come from two sensors on Kinect: the RGB camera and the time of flight (ToF) depth sensor. In turn, these produce three streams of vision input: a stream for RGB, a stream for depth field, and a stream for tracked human body rigs. These data streams have different rates and resolutions.

Tracking object: We used the Glyph detection algorithm (Kirillov 2016), with some adjustments to detect Glyph markers stuck on the objects (Glyph markers are black and white checked square). Markers put on different objects are distinguished to simplify the tracking process. For frames where tracking is lost, the marker’s 2D position is interpolated. 2-dimensional data is projected into 3-dimensional by using depth field. Body rig joint points are already tracked by Kinect’s SDK.

Normalizing rate: Different streams of data were regenerated with the same rate by re-sampling with interpolation. We used the rate of 24fps, which is the same as the RGB stream.

Quantitative features

3D features are generated by the following methods:

• Relative motions between different objects are approximated by calculating the distance vectors among these entities.

• To model human body rigs, vectors among the following points are calculated: middle point between the shoulders, left hand tip and right hand tip.

• To model objects, vectors between two diagonally opposing points are used.

Embedded 2D features

For each factor model, we used Principle Component Analysis (PCA) to project points considered into 2-dimensional planes, with the hope that the kept dimensions will keep the maximum variation, while provide an efficient way to visualize and reason about the data. The set of features for each frame is analogous to the 3D case, but with all data points from each factor model projected through PCA.

Frame-level qualitative features

The set of qualitative features are downstream features from the set of quantitative features. We employed 4 feature types as listed before. Visualizing trajectories of objects in embedded spaces gave us insight into the quality of our extracted features. For example, we observed that 2D qualitative trajectory calculus has a strong explanatory power in distinction of inherent motion of objects. For example, Figure 4a shows typical trajectories of two corner points of a rolling object. Direction between these points shows periodical change that is easily captured as a change of cardinal directions in the feature space.

(a) Corners of a rolling object (b) Hands in relative to body Figure 4: Samples of trajectories in embedded spaces

Event-level qualitative features A downstream set of event features would require a method to summarize the change of frame-level features across the event duration. There are multiple ways to do that, one is by specifying a set of primitive actions, such as in (Suchan, Bhatt, and Santos 2013). This approach requires a hierarchical fuzzy decomposition of event into subevents.

Another approach that employs a form of Inductive Logic Programming (ILP) is that of (Dubba et al. 2015). This approach is based on Inductive-Abductive reasoning framework, in which the authors look for basic, minimal, constraint satisfied narratives that explain changes of RCCstates as observed from event captures for each event type. This framework is quite interesting and might be effective, especially when event interpretation depends only on change of RCC-states (dc, touch, in). We, however, are not sure how to adapt this framework for other spatial qualitative relationships, because that would requires generation of intermediate qualitative states for each object pair. For an ordered set of states in RCC, it might be feasible, but for cardinal directions or QTC states, the path between two states is not unique, and the number of intermediate states to be considered would increase quickly. For a small set of training data, that might lead to a form of overfitting, as too many rules are produced.

For this reason, we resort to use a simple and featurebased only representation. In fact, we use the frame-level features of the first frame and the last frame, plus the different vector between these two.

Multilayer perceptron learning (MLP)

For event-level features, we use a simple multilayer perceptron model for comparison with our sequential neural network models. The multilayer perceptron (MLP) is one of the earliest neural network model, employing a feedforward infrastructure with backpropagation update. Here we use a rectified MLP (a stack of layers in which each has a linear layer combined with a Rectified Linear Unit (Glorot, Bordes, and Bengio 2011)). Dropout is also applied to reduce overfitting.

Long-short term memory (LSTM)

LSTM is a flavor of deep Recursive neural network (RNN) that has generally solved the problem of “vanishing gradients” in traditional RNN learning (Hochreiter and Schmidhuber 1997; Schmidhuber 2015) and has found their application in a wide range of problems involving sequential learning, such as hand written recognition, speech recognition, gesture recognition, etc.

Figure 5: LSTM model with possible constraints of outputs with CRF. CRF layer is represented as dashed links among predicted labels.

We will not describe in detail the LSTM model, as it has become a standard sequential learning method. Interested readers are recommended to take a look at either our implementation 1 or a popular reference on RNN and LSTM 2. Briefly, however, the model passes each feature vector through a linear layer before feeding each sequence into an LSTM. Each label requires a separate LSTM cell, . Output of each LSTM cell is a term corresponding to 5 semantic slots in the tuple. We will combine these values with our CRF weights, discussed in LSTM-CRF.

Conditional Random Field (CRF)

Figure 6: Reformation from general CRF (left) to TreeCRF (right)

CRF has been used extensively to learn structured output as it allows specification of constraints of output labels (Sutton and McCallum 2006). In this model we wish to constrain the outputs so that: one object (Performer or the other objects) is not allowed to fill two different syntactic slots; when there is no verb, all the other slots should be None; locative and preposition are dependent, because if locative is None, preposition must also be None and vice versa. The edges between nodes on the left side of Figure 6 show the dependencies on output labels that we wish to model.

However, training and classifying using a full CRF model would be more difficult, especially when implemented with a neural network architecture. We modified the model into a tree-CRF structure (right side of Figure 6) to make the model learnable using dynamic programming. The complexity of the algorithm reduced from to where n is the size of our vocabulary. The learning problem is thereby changed to learning the weights along the edges on the treeCRF, for example, P locativepreposition (together with parameters of LSTM). The directionality of edges is the forward direction of the message passing algorithm used for learning (and in reverse, for testing using the backward direction).

Use of CRF as a constraint layer Naturally, as we have learned 5 models (either 5 MLP or 5 LSTM) to predict 5 values in the output, we want to regularize the output combination by CRF. Moreover, LSTM-CRF is a natural extension of LSTM applied for constrained outputs. For instance it is used for named entities recognition task to model constraints on BIO labels (Huang, Xu, and Yu 2015).

To put CRF learning on top of MLP or LSTM, we modify the term t (the term before softmax) produced by outputs of MLP or LSTM as followings.

where l, s, o, p, v stand for Locative, Subject, Object, Preposition and Verb respectively.

In training, softmax is calculated for a predicted label combination, namely as below. We can calculate the log of sum using message passing over the tree nodes of the CRF tree. We use cross entropy between predicted distribution and correct output as the cost in training.

In evaluation, a similar message passing algorithm is used, but instead of logsum, we use max to calculate the probabilities and argmax to keep track of the best combination.

Figure 7: Event capture with fine-grained annotation

To demonstrate our model’s capability to learn the spatio-temporal dynamics of object interactions in events, we use a collection of four action types: push, pull, slide, and roll, along with three different spatial prepositions used for space configurations between objects, namely toward (when the trajectory of a moving object is straightly lined up with a destination static object and makes it closer to that target), away from (makes it further from that object) and past (moving object getting closer to static object then further again).

Afterwards, for each session, we sliced the events into short segments of 20 frames. Two annotators were assigned to watch and annotate them (segments can be played back). To speed up annotation, only event types related to original captured types are shown for selection. For instance, if the event type of the captured session is “The performer pushes A toward B”, other available event types are “The performer pushes A”, “A slides toward B” or “None”.

We explored different combinations of hyperparameters for our MLP and LSTM models. Two methods are used to combat over-fitting: (i) dropout (Hinton et al. 2012) (for LSTM this a dropout wrapper on LSTM cell, for MLP this is a dropout wrapper on each layer) and (ii) gradient clipping by a global norm. Following is the list of hyperparameters we used for tuning: • Number of LSTM layers: [1, 2] • Size of hidden layer: [200, 400] • Learning rate: [0.05, 0.1, 0.2, 0.5] • Dropout rate: [0.5, 0.6, 0.8] • Learning rate decay: [0.94, 0.95, 0.96] The network is trained with mini-batch gradient descent optimization for 200 epochs (LSTM) or 500 epochs (MLP) on the Tensorflow library.

Captured sessions are split for 5-fold cross validation, i.e., 24 sessions for training and 6 for testing on each fold. We use the LSTM-CRF model with raw input data as the baseline. A prediction is correct if all slots are correct. Performance is reported in the following tables:

Table 1: Evaluation

Table 2: Label precision breakdown for Frame-2D-Qual-LSTM-CRF. Note that because the labels are not independent, precision of the joint model is not the product of individual figures

We can observe a significant improvement of classifica-tion using 2-dimensional frame-level qualitative features. Frame-level quantitative features did improve over our baseline, but the improvement is not as impressive. Moreover, summarizing frame-level features to create event-level features creates a lossy representation that is not be able to learned efficiently.

Given these results, it is worth considering possible explanations for our findings. Firstly, as pointed out by (Yang and Webb 2009) and (Jiang et al. 2017), qualitative representation is a method of discretization, which makes data sparser, therefore easier to learn. Especially when taking the difference between features of two adjacent frames, as a qualitative feature strongly distinguishes between 0 and 1, the effect of sequential change is more pronounced.

Moreover, we observed that the best performance for Qual-LSTM-CRF is achieved by configuring two layers of LSTM with 400 nodes on the hidden layers, while for other models, the number of layers does not affect the performance significantly. Differently from a feed-forward neural network such as Convolutional Neural Network (CNN), which can learn more abstract and useful features when it get deeper, LSTM needs some help from the representation of features to reap a benefit from going deeper.

We also learned that a simple summary representation for events is not effective. We were very surprised that the results from event level features were much lower than we expected. Though we did expect that the loss of information regarding path contour could make the classifier indifferent to the distinction between rolling and sliding, that alone might not be able to explain the bad performance. Another possible factor is that there are many frames where the tracking is either lost or very noisy that the representation for a single frame is meaningless. If it happens to be the beginning or end frame, the representation is not of high quality. Using a sequence of frame representation has a positive effect to compensate for frame noise.

If instead we also takes into account features in intermediate frames, the representation comes back to a sequential one. There may be a better way to do this, such as summarizing for each feature separately, or using the InductiveAbductive framework as mentioned before, but as we already discussed, it is unclear how to perform these steps efficiently. This leads us to a belief that possibly a representation, which is sparse for each point in time but dense temporally, would be reasonably easy to learn, resulting in more compact and explanatory models.

In this study we have explored a wide range of feature extraction methods to learn what is the best way to represent events for classification. Even though our event domain is quite restricted or artificial, it represents complex human-object interactions, and therefore we believe that our conclusion could be generalized to other domains of similar complexity, such as that of human-robot interaction.

We acknowledge our shortcoming in the investigation of event-level qualitative features. More complex methods, such as those in (Dubba et al. 2015) could be used in a similar manner, and we will leave that for future directions.

One more dimension that we wanted to explore but we have not is that of the embedding method from 3-dimensional to 2-dimensional data. Methods other than PCA could be used, such as multidimensional scaling. We will also leave that for future research.

We have started to extend this learning framework to a more generic human-object problem in which objects are of common types, such as balls, books etc, requiring a subtask of classifying objects involved in an interaction as well. With more complex variation in shape of objects in movements, we expect to deal with more difficulty in object model representation. We will base on another work that our lab has developed called VoxSim (Krishnaswamy and Pustejovsky 2016) to scaffold an estimated representation for each object. Similar to the way that we have modeled internal movements of blocks in our Block World by qualitative change among markers’ corners, movements of generic objects could be estimated by considering relative movements between their components. For example, movement of a cup could be estimated by observing the relative motion between its rim and its handle. In turn, these parts could be estimated using both visual features and the cup’s Gibsonian affordances when in interaction with performers.

Based on VoxSim simulated environment, we are also bootstrapping a dataset to be used as auxiliary training samples. The idea is that training data for our generic event clas-sification suffer from sparseness with regards to all different positions and configurations of objects. We intend to use our environment to create a multitude of event simulations, projecting them into a same embedded space as our visual captures. We are just at the very beginning phase of doing that, and we hope that more results and discussions would be followed in the near future.

Figure 8: “Put an apple on a plate” - a generic event realization in our embedded simulated environment. From (Krishnaswamy and Pustejovsky 2016)

This work is supported by a contract with the US Defense Advanced Research Projects Agency (DARPA), Contract W911NF-15-C-0238. Approved for Public Release, Distribution Unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. We would like to thank Nikhil Krishnaswamy and Keigh Rim for their discussion and input on this topic. All errors and mistakes are, of course, the responsibilities of the authors..

Allen, J. 1984. Towards a general theory of action and time. Arificial Intelligence 23:123–154.

Andrew, U.; Mark, D.; and White, D. 1991. Qualitative spatial reasoning about cardinal directions. In Proc. of the 7th Austrian Conf. on Artificial Intelligence. Baltimore: Morgan Kaufmann, 157–167.

Delafontaine, M.; Cohn, A. G.; and Van de Weghe, N. 2011. Implementing a qualitative calculus to analyse moving point objects. Expert Systems with Applications 38(5):5187–5196.

Do, T., and Pustejovsky, J. 2017. Fine-grained event learning of human-object interaction with lstm-crf. Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN).

Do, T.; Krishnaswamy, N.; and Pustejovsky, J. 2016. Ecat: Event capture annotation tool. Proceedings of ISA-12: International Workshop on Semantic Annotation.

Dougherty, J.; Kohavi, R.; Sahami, M.; et al. 1995. Supervised and unsupervised discretization of continuous features. In Machine learning: proceedings of the twelfth international conference, volume 12, 194–202.

Dubba, K. S. R.; Cohn, A. G.; Hogg, D. C.; Bhatt, M.; and Dylla, F. 2015. Learning relational event models from video. Journal of Artificial Intelligence Research 53:41–90.

Gatsoulis, Y.; Alomari, M.; Burbridge, C.; Dondrup, C.; Duckworth, P.; Lightbody, P.; Hanheide, M.; Hawes, N.; and Cohn, A. 2016. Qsrlib: a software library for online acquisition of qualitative spatial relations from video. In Workshop on Qualitative Reasoning (QR16), at IJCAI-16.

Glorot, X.; Bordes, A.; and Bengio, Y. 2011. Deep sparse rectifier neural networks. In Aistats, volume 15, 275.

Hinton, G. E.; Srivastava, N.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. R. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580.

Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. Neural computation 9(8):1735–1780.

Hoogs, A., and Perera, A. A. 2008. Video activity recognition in the real world. In AAAI, 1551–1554.

Huang, Z.; Xu, W.; and Yu, K. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.

Jiang, X.; N de Souza, E.; Liu, X.; Haji Soleimani, B.; Wang, X.; L. Silver, D.; and Matwin, S. 2017. Partitionwise recurrent neural networks for point-based ais trajectory classification. In 25th European Symposium on Artificial Neural Networks, Computtional Intelligence and Machine Learning, 529–534. ESANN.

Kirillov, A. 2016. From glyph recognition to augmented reality. https://www. codeproject.com/articles/258856/

from-glyph-recognition-to-augmented-reality. Accessed: 2016-03-15.

Krishnaswamy, N., and Pustejovsky, J. 2016. Voxsim: A visual platform for modeling motion language. In COLING (Demos), 54–58.

Le, Q. V.; Zou, W. Y.; Yeung, S. Y.; and Ng, A. Y. 2011. Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 3361–3368. IEEE.

Li, W., and Fritz, M. 2016. Recognition of ongoing complex activities by sequence prediction over a hierarchical label space. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, 1–9. IEEE.

Mavridis, N.; Bellotto, N.; Iliopoulos, K.; and Van de Weghe, N. 2015. Qtc 3d: Extending the qualitative trajectory calculus to three dimensions. Information Sciences 322:20–30.

Pustejovsky, J.; Lee, K.; Bunt, H.; and Romary, L. 2011. Iso-timeml: A standard for annotating temporal information in language. In LREC.

Pustejovsky, J. 1995. The Generative Lexicon. Cambridge, MA: MIT Press.

Rohrbach, M.; Amin, S.; Andriluka, M.; and Schiele, B. 2012. A database for fine grained activity detection of cooking activities. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, 1194–1201. IEEE.

Roy, A., and Pal, S. K. 2003. Fuzzy discretization of feature space for a rough set classifier. Pattern Recognition Letters 24(6):895–902.

Ryoo, M. S., and Aggarwal, J. 2010. Ut-interaction dataset, icpr contest on semantic description of human activities (sdha). In IEEE International Conference on Pattern Recognition Workshops, volume 2, 4.

Sabharwal, C. L., and Leopold, J. L. 2014. Modeling cardinal direction relations in 3d for qualitative spatial reasoning. In Mining Intelligence and Knowledge Exploration. Springer. 199–214.

Schmidhuber, J. 2015. Deep learning in neural networks: An overview. Neural Networks 61:85–117.

Shahroudy, A.; Liu, J.; Ng, T.-T.; and Wang, G. 2016. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. arXiv preprint arXiv:1604.02808.

Suchan, J.; Bhatt, M.; and Santos, P. E. 2013. Perceptual narratives of space and motion for activity interpretation. In 27th International Workshop on Qualitative Reasoning, 32. Sutton, C., and McCallum, A. 2006. An introduction to conditional random fields for relational learning, volume 2. Introduction to statistical relational learning. MIT Press.

Tulving, E. 1983. Elements of Episodic Memory. Oxford University Press.

Veeraraghavan, H.; Papanikolopoulos, N.; and Schrater, P. 2007. Learning dynamic event descriptions in image sequences. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, 1–6. IEEE.

Yang, Y., and Webb, G. I. 2009. Discretization for naivebayes learning: managing discretization bias and variance. Machine learning 74(1):39–74.