Event recognition, which is the classification and re-identification of relevant events over time, has a long history (F¨ul¨op et al. 2010; Sakaki, Okazaki, and Matsuo 2010; Wang, Hovy, and Dredze 2015; Doornik 2009; Suprem and Pu 2019; Musaev, Wang, and Pu 2014). Event recognition comprises of two intertwined processes: (i) data processing and (ii) event detection, that have co-dependence: event detection requires processing raw real-world data to extract relevant signals, and useful data processing requires knowing which events to follow in the universe of events (Figure 1). There have been various works on data processing, event detection, and complex event detection (F¨ul¨op et al. 2010); furthermore, emergence of powerful compute resources
Figure 1: Co-dependence of Data Processing and Event Detection. Data Processing identifies relevant signals in probabilistic sources. Event Detection on corroborative sources fine-tunes Data Pro-cessing for robustness against drift
combined with large volume streaming data has led to recent works on event detection with stream processing (Chuanfei et al. 2010) and machine learning (Suprem and Pu 2019). More recently, there have been works on exploiting human sensors from social media sources for dense global physical event recognition, such as earthquake (Sakaki, Okazaki, and Matsuo 2010), hurricane (Wang, Hovy, and Dredze 2015), and landslide detection (Musaev, Wang, and Pu 2014).
A common challenge in event recognition is the concept drift phenomenon, where the distribution of data changes continuously. The concept drift phenomenon is
Figure 2: Drift in signals for an event over time. The signals composing an event change due to different types of concept drift. For example, due to cyclical drift, Signal 4 disappears at and reappears at
.
well-documented in event recognition (Gama et al. 2014; Gama et al. 2004). Under concept drift, static event detection methods (rule-based, machine learning, or deep learning methods) exhibit performance degradation over time. So far, effective concept drift adaptation strategies for noisy social media sources data remains unexplored; most work in drift detection and adaptation focuses on closed, small datasets with well known drift attributes (Scheirer et al. 2012).
In this paper, we present EventMapper , a framework for event recognition that exploits the co-dependence of data processing and event detection to support long-term event recognition that can adapt to concept drift in social media sources. EventMapper integrates two types of sources: high-confidence corroborative sources and low-confidence probabilistic sources. Corroborative Sources. A high-confidence corroborative source is a dedicated physical or web-based sensor that provides annotated physical event information based on experts. Due to expert corroboration, corroborative sources have reduced noise and drift (De Albuquerque et al. 2015). However, corroboration increases their cost; so corroborative sources have reduced coverage and higher latency. Probabilistic Sources. A probabilistic source is any source without corroboration, such as raw web streams or human sensors. Lack of corroboration makes such streams, such as Twitter and Facebook, noisy and drifting (Ritter et al. 2012). However, such sources are globally available and have low latency (De Albuquerque et al. 2015).
The EventMapper framework allows deployment of systems for weak-signal event recognition. In contrast to strong-signal events such as earthquakes and hurricanes, which have hundreds of thousands of corroborative and probabilistic signals per event, weak-signal events have 1-3 signals per event. Strong-signal events have easily separable signals and detection can be per-formed retrospectively with trend analysis (Sakaki, Okazaki, and Matsuo 2010; Wang, Hovy, and Dredze 2015). Conversely, weak-signal event detection is more difficult since signals are not easily separable. Since each event has few signals, they are hidden in noise and drift and require precise data processing with statistical and machine learning methods.
By integrating corroborative and probabilistic sources by exploiting their co-dependence, EventMapper improves upon static event recognition for weak-signal events: (i) corroborative sources, which are used directly for event detection, fine-tune the data processing modules for social sensors; consequently, data processing modules remain robust
Figure 3: ML Classifier deterioration We compare statistical and deep models over time on our data (described in Applications section). After initial training, models are evaluated on subsequent months (M1, M2, M3).
to concept drift over time, and (ii) data processing modules use statistical and machine learning methods to extract relevant signals from probabilistic sources for event detection. Our contributions are as follows: 1. We present the EventMapper framework; we will be releasing the code to the open-source community. The EventMapper framework supports weak-signal event detection while remaining ro-bust to concept drift. By exploiting the co-dependence between data processing and event detection, EventMapper automates concept drift adaptation. 2. We describe three applications deployed on the EventMapper framework for three weak-signal events: landslide detection, wildfire detection, and flooding detection. For each application, we show evidence of that weak-signal nature of the events. We will provide links to the application demos.
Events and Signals The concept drift phenomenon is well known in event detection (Gama et al. 2014; Gama et al. 2004). Under concept drift, the signals characterizing an event change over time as new signals are introduced. Also, existing signals may disappear. With desired event in S, the social media stream, we aim to detect it from stream elements, e.g. a post written in natural language and decomposed into numeric signals s with w2v (Rong 2014) for event detection. When new words enter the vocabulary, new signals are introduced. Furthermore, recent approaches in NLP use surrounding context of words to create a word context vector (Kim 2014); as a word’s definition changes due to memes or viral content, the context vector also changes. We characterize concept drift in terms of events and signals. Each post
in the stream S is a distribution over the events
, which includes
. Each post is also a generative model over the component signals
. So,
where k is the number of signals present in the stream; k changes as new words are added, and existing words become
Figure 4: EventMapper The EventMapper dataflow integrates corroborative and probabilistic sources; corroborative sources provide ground truth event detection, and probabilistic sources provide real-time, global coverage.
obsolete. Changes in word meanings also change is the strength of a signal
in the desired event.
Concept Drift
Concept drift occurs when the distribution of and
change for a desired event
(Figure 2). Since an ML classifier learns a projection from the signals domain to the classification domain (e.g. binary classification), changes in the signals domain due to drift makes the learned projection invalid. Under drift, a static ML classifier will exhibit performance degradation over time. We show this in Fig- ure 3, where we used static classifiers trained on landslide data from 2014 (see Implementation section on data collection steps) to evaluate performance on 2017, 2018, and 2019. In each case, performance degrades over time due to drift in the online data stream and increasing social media noise that renders older models obsolete. Several variants of drift are known: (i) gradual drift slowly changes the coeffi-cients of existing signals and adds new signals, with lexical diffusion (Eisenstein et al. 2014) as a representative example; (ii) in contrast, sudden drift causes rapid changes in co-efficients, with viral memes as a representative example; (iii) cyclic drift re-introduces signals periodically before disappearing them, e.g. with landslide detection, our system needs to be aware of election landslide related posts each October and November in the US; and (iv) flash drift introduces new meanings for a short time.
Some challenges particular to the streaming domain include: difficulty of concept drift, natural language processing (NLP) on short text data, and weak-signal event detection. We introduce related work for each area and tie them back to event recognition in the streaming domain.
Concept Drift
Recent works have focused on adapting the static classi-fiers to the dynamic, streaming domain. However, these approaches keep many of the data assumptions of the static models; a thorough survey is available in (Gama et al. 2014; Gama et al. 2004). Assumptions. Under the closed data assumption, streaming data is well specified by the training data and large amounts of ground truth labels can be quickly generated for model updates. Under the immediate feedback assumption, oracle labels are available for drift updates; recent approaches have relied on weak supervision in lieu of oracles (Suprem, Musaev, and Pu 2019).
The approach in (Ren et al. 2018) uses several drift detector modules combined with updates to create an adaptive ensemble. Windowing is used in (Bifet and Gavalda 2007) to track concept drift and record irrelevant data to retroactively fix prior cloassifications. The KNN-based approach in (Losing, Hammer, and Wersing 2016) uses nearest neighbor search to identify the best window of models for a sample. Finally, (Gama et al. 2004; Baena-Garcıa et al. 2006) use explicit error tracking to detect drift.
Concept drift in online streams has seen limited investigation. Recent works include (Sakamoto et al. 2015), which explore complex event detection in the presence of sensor drift. Further, (Eisenstein et al. 2014) explore lexical diffusion, an example of gradual drift in signals and coefficients based on geographical location.
Many of the existing approaches in event detection also assume data without concept drift. Such assumptions, which were made in Google Flu Trends (GFT), create models that degrade over time. GFT was originally created to complement the CDCs flu tracking efforts by identifying seasonal trends in the flu season (Doornik 2009). Failure to account for seasonal changes in event characteristics led to increasing errors over the years, and by 2013, GFT missed the trends by 140%. This error has been attributed to exclusion of new data from CDC, changes in the underlying search data distribution itself, and cyclical data artifacts (Lum and Isaac 2016; Lazer and Kennedy 2015; Kugler 2016). Short Text Streams. NLP plays a key role in extracting useful signals from text. However, NLP is suited for longtext data; since social media text is short and noise (Ritter et al. 2012), traditional NLP techniques are not sufficient for classification (Sriram et al. 2010). Weak-Signal Event Detection. Event recognition on web streams have primarily focused on strong-signal events, such as earthquakes (Sakaki, Okazaki, and Matsuo 2010), flu (Doornik 2009), and hurricans (Wang, Hovy, and Dredze
Figure 5: Irrelevant Events Percentage irrelevant events for each event type (landslide, wildfire, flooding). Over 60% events from keyword stream are irrelevant to event type. These irrelevant items remain after we use keyword filtering (e.g. using the landslide keyword) to download social media posts and static stopwords to filter out some items (e.g. election to filter out election results).
2015). These are large signal events since cases can be veri-fied and have abundant reputable data. We focus on weak-signal events that have little to no corroboration (for our events detected on social media, only 5% of events have corroboration from reputable sources; the rest are classified using our system). Specifically, we focus on landslides, wild-fires, and flooding detection.
We now describe our EventMapper framework. We first cover the framework dataflow at a high level. We then cover the probabilistic source event detection that is integral to EventMapper . Finally, we provide salient implementation details.
EventMapper Dataflow
The EventMapper framework, shown in Figure 4, integrates corroborative and probabilistic sources for dense, global, real-time physical event recognition. Ground truth events detected from corroborative sources are used to fine-tune data processing (which includes data cleaning, metadata extraction, and ML models for classification) for probabilistic sources. Continuous fine-tuning requires concept drift adaptation, which means updating data processing modules with the current stream’s distribution. Current approaches described in Related Work perform this update manually and in the closed dataset domain. EventMapper ’s advantage is in automating the continuous fine-tuning, allowing scalable drift adaptation that remains functional long after initial model construction. This allows the data processing steps to remain robust to concept drift.
Corroborative Source. The corroborative sources provide ground truth events. Since corroborative sources use multiple sources and human experts, they are slower and have low coverage. For example, news coverage about landslides appear 2-3 days after the event has occurred, which renders event detection based on a news article irrelevant since the
Figure 6: Dynamic Stopword Filtering Percentage irrelevant posts filtered by dynamically updated stopword list. Between 10-20% of irrelevant items are filtered out this way. The remaining are procesed with more fine-tuned ML models.
delay between event and detection is too great. News articles also do not have dense global coverage, since smaller landslides in rural areas may not be reported. Probabilistic Source. The probabilistic sources are derived from social media sources. Since probabilistic sources represent a large variety of events, data processing is required to identify relevant signals for event detection. In contrast to corroborative sources, where each source represents a spe-cific event (e.g. NASA MODIS detects wildfires only), a probabilistic source such as a Twitter stream covers multiple events. Even with keyword-based streams, the presence of lexical diffusion, multiple word meanings (polysemy) and memes increase noise. For example, landslide detection on Twitter by following tweets using the word landslide or mudslide also returns tweets for election landslides, the ice cream Mississippi Mudslide, and the song Landslide by Fleetwood Mac. Furthermore, the instances of tweets relevant to the landslide disaster are dwarfed by irrelevant tweets, see Figure 5. Data Cleaning. Probabilistic streams are noisy and need data cleaning before analysis. Cleaning can remove especially noisy examples. EventMapper uses stopwords to perform data cleaning on probabilistic sources. In contrast to conventional approaches which create stopwords of common English terms, EventMapper uses prior irrelevant data to continuously update the stopwords list.
EventMapper keeps track of irrelevant posts from probabilistic sources detected by the Event Detection module (see Figure 4. Periodically, most frequent terms in the irrelevant posts are added to the stopword list and old terms in the list are pruned. Specifically, the top-k most frequent terms in the irrelevant posts are added to the stopword list, replacing the prior frequent stopwords list; we let k = 5. With dynamic data cleaning, EventMapper is able to filter out between 10-20% of irrelevant posts without requiring ML classifiers, see Figure 6. This has an important advantage: earlier filtering reduces burden on the event processing system since irrelevant posts that are filtered out do not use up valuable compute resources.
Metadata Extraction. While traditional methods have used only the raw text for detection, more recent methods have exploited surrounding metadata to improve event detection or classification. For example (Hemmings-Jarrett, Jarrett, and Blake 2018) uses metadata from users to improve sentiment classification, and (Popat et al. 2018) uses metadata to determine user and information credibility. In EventMapper , metadata extraction is similarly used to augment the raw text from the stream. Each event may require different metadata, and extraction is left to the framework deployer. In our landslide, flooding, and wildfire events, we primarily perform location extraction.
However, only 0.5% of social media posts have a geotag, necessitating location extraction from the text content as well. Social sensor data has high noise, and Named Entity Recognition (NER), which is used for location extraction from text, often fails (Middleton et al. 2018) and misses many locations present in a post’s text content. Because EventMapper exploits the co-dependence between the corroborative sources and probabilistic sources, we use prior events detected in both dataflows to improve location extraction.
Prior events detected from the corroborative source dataflow have a location associated with them, since corroborative sources provide locations. The locations are then used for substring matches in the metadata extraction. As new events are detected, their locations augment the sub-string match list. Any locations matched with substring match are used to continuously train a NER using the corresponding probabilistic source text. In effect, our EventMapper ’s applications combine three location extractors: off-the-shelf NER, continuously trained NER, and sub-string matches. Corroborative Integration. Each real-world physical event has two primary attributes: location of event and time of occurrence. Since these spatio-temporal attributes are common to most physical event types we want to detect, EventMapper takes advantage of these to further tune ML classifiers. Each application deployed on EventMapper has a collection of ML classifiers that take as input a text post with its metadata and provide as output a label of relevance or irrelevance (binary classification). Relevant posts are event classifications that will be shown to end users, and irrelevant posts are used for updating the stopword filters. However, as we showed in Related Work and Figure 3, static ML clas-sifiers exhibit performance deterioration over time. So, we need to update the ML classifiers over time.
Classifier updates require training data; recent approaches have used active learning to reduce the number of training data, and therefore, labeling cost. However, this is not scalable for streaming web data. EventMapper solves this with Corroborative Integration: by mapping both probabilistic posts with metadata and corroborative events to the same spatio-temporal grid, EventMapper can automatically label some data points to create training data. If a corroborative event occurs at the same time and place as a probabilistic post with the event keywords, there is a high likelihood the social media post is relevant to the event. In effect, we perform weakly supervised labeling with the corroborative in-
Figure 7: Event Detection We perform event detection using ML classifiers for probabilistic source posts that could not be labeled with corroborative integration. Fine-tuned classifiers are selected for each sample and a dynamic ensemble is constructed to label the sample.
tegration step. These weakly supervised labels can then be used as training data to update ML classifiers. While only 5% of the probabilistic source data can be labeled with corroborative integration, it is sufficient to perform ML classi-fier updates, as we show in Results. Negatively Labeled Data. One limitation of corroborative integration is only positive labels can be assigned to probabilistic posts. Negative labels cannot be assigned since probabilistic posts outside corroborative source coverage represent are of unknown relevancy because corroborative sources are delayed and have low coverage.
In EventMapper , we use stopword filtered posts as negative samples. Since the stopwords within the text themselves present a strong signal to ML classifier and can skew it to perform the same function as a stopword filter, we remove the term from the post without replacement. So given a sentence of word tokens where
is the stopword that triggered the stopword-filtering, we use the derived sentence
as a negative label, removing
. During model updates or generation, positive and negative samples are converted to word embeddings with w2v for model training.
Probabilistic Source Event Detection
Event Detection. Since corroborative sources provide ground truth events, we focus on event detection in the probabilistic source dataflow. The Event Detection module, shown in Figure 7, performs two steps: (i) Update and (ii) Classification.
Update Step. EventMapper uses the labeled data from corroborative integration to update existing classifiers. First, existing classifiers are evaluated on the labeled data with explicit drift detection using EDDM (Baena-Garcıa et al. 2006) (see Related Work). Classifiers with low performance can be pruned from the set of classifiers or flagged for update with the labeled data, based on user preferences. In our application implementations, we retain low performing models to archive them and use a copy of the model for updates. We then perform unsupervised concept drift detection to identify changes in the data distribution. We use the method from (Suprem and Pu 2019; Suprem 2019) to detect drift using high density bands. EventMapper maintains a memory of streaming points. We compare the distribution of the high density bands to the distribution of the streaming data using the Kullback-Leibler divergence. If drift is detected with the approach in (Suprem 2019), EventMapper uses the training data from corroborative integration to create new ML clas-sifiers. For each updated or generated classifier, EventMapper stores the classifier along with the training data for the classifier. The training data is indexed by the mean of the training data.
Classification Step. For each post from the probabilistic stream that could not be labeled with corroborative integration, EventMapper classifies its relevance with the ML clas-sifiers generated in the Update Step. While some approaches have used all stored models as an ensemble, EventMapper dynamically creates an ensemble to best fit a data point. Given a probabilistic post (data point), EventMapper sorts all classifiers on their training data’s distance to the post. We measure distance using the train-ing data mean. The knearest classifiers are used to create an ensemble, with each classifier weighted by its distance to the data point. The dynamically created ensemble is used for relevancy prediction.
EventMapper Implementation
We now describe the implementation for EventMapper . The framework is designed for extensibility and allows us to deploy applications for different event detectors quickly. Each operation in the EventMapper dataflow (Figure 4) is a process primitive. Instead of a linear dataflow with data passed between processes, EventMapper uses a pub/sub interface to decouple process primitives. This accomplishes two things: (i) each process can be updated and managed independently, and (ii) the application remains fault-tolerant to crashes in any one process.
Streamers. EventMapper provides built-in streamers for probabilistic sources such as Twitter and Facebook. Our current work includes integration of YouTube and Instagram video and image streams as well. Streamers operate on user-defined, generic keywords for an event type provided as a configuration. In our applications, we use the following keywords for the streamers: • Landslide: landslide, mudslide, rockslide • Wildfire: wildfire, brushfire • Flooding: flood, heavy rain Since these are generic keywords, they include significant
noise. Fine-tuning the keywords themselves can reduce our
coverage and is not scalable, since drift may occur. So, we rely on EventMapper s fine-tuning to filter posts for event recognition.
Pub/Sub Interface. . EventMapper uses a pub/sub interface to decouple processes. We use Apache Kafka as the pub/sub backend. Each process publishes its message to a Kafka topic, and subsequent processes susbcribe to the topic to receive the message. To ensure the same data point is not read twice during process crashes, we need to store read status for messages. Since Kafka is a minimal interface without control over read/write status of messages for expiration, we manage message read status with a Redis key value store.
Each process in EventMapper has an import key and export key, which are unique strings for each process. The import and export keys function both as pub/sub topics and Redis keys. An EventMapper primitive process subscribes to its import key and publishes to its export key. Apache Kafka does not have support for recording message read/write status, so we use Redis to manage this. For each message, EventMapper updates on Redis the read/write values for its associated import key; it records message ID (a unique ID for each post), message offset, and partition. The offset and partition are used to recover an applicatio’s position in the stream after a process crash. This also allows us to build many-to-many, one-to-many, and many-to-one dataflows in addition to the traditional one-to-one. Since each process manages its own import keys offset with Redis, a message is guaranteed to be read onloy once by each process. Process Management. EventMapper manages each process to ensure continuous operation of an event detection application. To reduce overhead, each process is deployed independently and records its own process ID. EventMapper checks process logs and process ID for non-operation, at which point any zombie executions are killed and the process is restarted. Subsequent failures in succession trigger an alert for the end-user to investigate process restart failures. Process Primitive. Each process in EventMapper extends a base process primitive class. This allows applications to have flexibility in signal extraction for each module (data cleaning, metadata extraction, etc) while leaving message processing to EventMapper . Therefore, while signal extraction logic needs to be written for each new event type, the process-to-process communication, fault-tolerance, and crash recovery are managed by EventMapper .
In this section, we provide application details for our three desired events: landslides, wildfires, and flooding. We will cover results in the next section.
Landslide Detection We select landslides as a desired event since they are a weak signal disaster with significant noise in social media streams; the use of the word landslide is polysemous, meaning it carries multiple meanings. The word landslide can refer to election landslides and a song Landslide (by Fleetwood Mac) in addition to the disaster landslide. Corroborative Sources. We use four corroborative sources for landslide detection: • NASA TRMM provides landslide likelihood data in select locations around the globe. TRMM has three levels of predictions: 1-day, 3-day, and 7-day. Each prediction level provides: landslide likelihood using NASA’s landslide models, closest location name, and latitude/ longitude. EventMapper uses location name to update substring match list for NER fine-tuning. We use the 1-day landslide predictions.
• USGS Earthquake provides detected earthquakes around the globe. For each instance, it provides magnitude and latitude/longitude of the epicenter. • NOAA GHCND provides daily rainfall data at NOAA weather stations around the globe. Each station provides its latitude and longitude, along with rain in the past day. Due to a combination of old equipment, budget cuts, and progressive expansion, many stations do not provide up-to-date information. • News articles about landslides provide late corroboration since they are delayed by 2-3 days. EventMapper follows articles with an off-the-shelf API (NewsAPI) with the landslide disaster tag. Locations are extracted with NER since articles are long text and NER succeeds on the structured news text (Middleton et al. 2018; Sriram et al. 2010). We use NASA TRMM and News as landslide ground-truth data, where available. We combine USGS and GHCND data, since heavy rainfall and earthquake in the same location indicates high probability of landslide (Musaev, Wang, and Pu 2014), to provide secondary ground-truth data. In the corroborative integration process, we map ground-truth events and probabilistic source posts to a spatio-temporal grid. For probabilistic posts at the same time and geographic location as ground-truth events, EventMapper labels them as relevant posts. The labeled posts from corroborative integration are used to fine-tune ML classifiers in the Event Detection process’ Update Step. Each probabilistic post that could not be labeled with corroborative integration is processed with the Classification Step.
Wildfire Detection
Wildfires have flared up to a greater degree over the past two years due to climate change (Stevens-Rumann et al. 2018). Furthermore, wildfires are expected to increase over the next years due to increased warming, longer fire seasons, increased emissions, and drier forests (Schoennagel et al. 2017; Liu, Stanturf, and Goodrick 2010). Representative examples include the ongoing (at this time) Australian wild-fires and the Amazon rainforest wildfires in 2018. Wildfires remain weak-signal however, since each wildfire instance is small and brush fires may crop up in isolated patches until it coalesces (Cruz et al. 2012). The term wildfire, like landslide, is also polysemous, since it can refer to pandemics and a Pokemon (children cartoon character). Corroborative Sources. We use three corroborative sources for wildfire detection. The primary corroborative sources are from NASA’s Fire Information for Resource Management System (FIRMS). • FIRMS MODIS provides wildfire detections from satellite canopy data (Levin and Heimowitz 2012). MODIS itself includes corroborative sources to clear its own false positives and improve fire detection. • FIRMS VIIRS (Lee et al. 2006) is an evolution of MODIS ; while VIIRS has fewer spectral bands than MODIS, it can read higher resolution of fire data and is
Figure 8: Weak-Signal Events Most events have 1-3 posts associated with them. The posts per event has a long tailed distribution; we show up to 10 posts per event for landslides, wildfires, and flooding events due to space.
more sensitive to fire radiance. However, it is not fully deployed at the moment, so we use both MODIS and VIIRS. • News articles about wildfires provide late corroboration. EventMapper uses NewsAPI to get articles with wildfire disaster tag. Locations are extracted with NER. MODIS and VIIRS provide ground truth events use to corroboratively label probabilistic source posts. Similar to landslide detection, labeled posts from corroborative integration fine-tune ML classifiers, which are in turn used for posts that could not be automatically labeled.
Flooding Detection
Similar to wildfires, flooding is expected to increase because of rising sea levels due to climate change (Pedruco et al. 2018). There have already been increased river floods in the past few years (Bevacqua et al. 2019). The keywords for flooding are also polysemous, since flood is used politically to refer to immigrants (Cervantes, Khokha, and Murray 1995), an antagonist in the Halo video game series, and in reference to economics (Carter and Sutch 2008). Corroborative Sources. We use the following two sources for flooding detection: 1. USGS Flood Gauge provides flooding information across the US. Each instance is provided with latitude and longitude, along with flooding magnitude such as major, moderate, minor, or no flooding. Out of 9298 current gauges, 3427 gauges (37%) are non-functional or not updated, reducing coverage. Also, the gauges are US-specific. 2. Similar to landslide and wildfires, we use NewsAPI to follow flooding tags to get late corroboration from news sources. The corroborative events are used for ML classifier fine-tuning. Classifiers are subsequently used for prediction.
We evaluate our applications deployed on the EventMapper framework. We have providence evidence of drift-based
Figure 9: Corroborative Integration Less than 5% of probabilistic source posts can be labeled with corroborative integration in each event type. We use these labeled posts for updating ML classifiers in the Classification Step of Event Detection. Y-Axis is log-scale.
performance deterioration in Figure 3. We first describe the weak-signal nature of the events.
Weak-Signal Events. As described in earlier, weak-signal events have few reports per event. Where corroborative event detection is not available, it is difficult to detect physical events from social media streams. We show evidence of the weak-signal nature of our events in Figure 8. For each event, we record the number of posts present in the event. More than 50% of events are associated with a single post, and 85% of events have 5 or less posts associated with them. This presents challenges: an ML classifier must be robust to noise to avoid flagging false positives. However, this may reduce global coverage by ignoring weak-signal events that are hidden in noise or surrounding large-signal events.
Event Detection. Since EventMapper continuously fine-tunes all stages of event recognition, from data cleaning (dynamic stopwords), metadata extraction (dynamic NER models), and classifiers (drift adaptive models), it is able to identify weak-signal events from 1-3 posts in addition to stronger-signal instances of weak-signal events (3% of events have more than 15 posts per event). Integrating cor-
Figure 10: Event Detection Landslide and wildfire detection applications in EventMapper showing detections from probabilistic source (Twitter) and corroborative integration (red markers). We omit flooding for space.
roborative and probabilistic sources allows EventMapper to take advantage of both source types: with ground-truth events from corroborative sources, EventMapper can continuously fine-tune all stages of an event detection pipeline; simultaneously, EventMapper ensures global coverage with probabilistic sources.
We show landslide and wildfire demo images in Fig- ure 10. For landslides, we show map with events detected from Twitter, Facebook, News, and multi-source integration (red markers). For wildfires, we show detected events on Twitter.
In this paper, we have proposed the EventMapper framework for event detection. The EventMapper framework is designed for weak-signal event detection. It accomplishes this by integrating corroborative and probabilistic sources to exploit the co-dependence of event processing on each source type. With corroborative sources, EventMapper can continuously fine-tune event processing on probabilistic sources. This allows for improved signal extraction for event detection, as we have shown: with continuous fine-tuning, we create a robust event detection pipeline that reduces long-term performance degradation. We also show EventMapper applications’ ability to detect weak-signal events. For each of our applications: landslides, wildfires, and flooding, EventMapper identifies events from 1-3 tweets. Compared to work in (Sakaki, Okazaki, and Matsuo 2010; Wang, Hovy, and Dredze 2015; Doornik 2009), which perform retroactive strong-signal detection on hundreds of thousands of posts, EventMapper performs real-time event detection.
[Baena-Garcıa et al. 2006] Baena-Garcıa, M.; del Campo- ´Avila, J.; Fidalgo, R.; Bifet, A.; Gavalda, R.; and MoralesBueno, R. 2006. Early drift detection method. In Fourth international workshop on knowledge discovery from data streams, volume 6, 77–86.
[Bevacqua et al. 2019] Bevacqua, E.; Maraun, D.; Vous- doukas, M.; Voukouvalas, E.; Vrac, M.; Mentaschi, L.; and Widmann, M. 2019. Higher probability of compound flooding from precipitation and storm surge in europe under anthropogenic climate change. Science advances 5(9):eaaw5531.
[Bifet and Gavalda 2007] Bifet, A., and Gavalda, R. 2007. Learning from time-changing data with adaptive windowing. In Proceedings of the 2007 SIAM international conference on data mining, 443–448. SIAM.
[Carter and Sutch 2008] Carter, S. B., and Sutch, R. 2008. Labor market flooding? migrant destination and wage change during americas age of mass migration. Migration and development within and across borders: Research and policy perspectives on internal and international migration 133–61.
[Cervantes, Khokha, and Murray 1995] Cervantes, N.; Khokha, S.; and Murray, B. 1995. Hate unleashed: Los angeles in the aftermath of proposition 187. Chicano-Latino L. Rev. 17:1.
[Chuanfei et al. 2010] Chuanfei, X.; Shukuan, L.; Lei, W.; and Jianzhong, Q. 2010. Complex event detection in probabilistic stream. In 2010 12th International Asia-Pacific Web Conference, 361–363. IEEE.
[Cruz et al. 2012] Cruz, M.; Sullivan, A.; Gould, J.; Sims, N.; Bannister, A.; Hollis, J.; and Hurley, R. 2012. Anatomy of a catastrophic wildfire: the black saturday kilmore east fire in victoria, australia. Forest Ecology and Management 284:269–285.
[De Albuquerque et al. 2015] De Albuquerque, J. P.; Herfort, B.; Brenning, A.; and Zipf, A. 2015. A geographic approach for combining social media and authoritative data towards identifying useful information for disaster management. International Journal of Geographical Information Science 29(4):667–689.
[Doornik 2009] Doornik, J. A. 2009. Improving the timeli- ness of data on influenza-like illnesses using google search data. Working paper.
[Eisenstein et al. 2014] Eisenstein, J.; O’Connor, B.; Smith, N. A.; and Xing, E. P. 2014. Diffusion of lexical change in social media. PloS one 9(11):e113114.
[F¨ul¨op et al. 2010] F¨ul¨op, L. J.; T´oth, G.; R´acz, R.; P´ancz´el, J.; Gergely, T.; Besz´edes, A.; and Farkas, L. 2010. Survey on complex event processing and predictive analytics. In Proceedings of the Fifth Balkan Conference in Informatics, 26–31. Citeseer.
[Gama et al. 2004] Gama, J.; Medas, P.; Castillo, G.; and Ro- drigues, P. 2004. Learning with drift detection. In Brazilian symposium on artificial intelligence, 286–295. Springer.
[Gama et al. 2014] Gama, J.; ˇZliobait˙e, I.; Bifet, A.; Pechenizkiy, M.; and Bouchachia, A. 2014. A survey on concept drift adaptation. ACM computing surveys (CSUR) 46(4):44.
[Hemmings-Jarrett, Jarrett, and Blake 2018] Hemmings- Jarrett, K.; Jarrett, J.; and Blake, M. B. 2018. (wksp) sentiment analysis of twitter samples that differentiates impact of user participation levels. In 2018 IEEE International Conference on Cognitive Computing (ICCC), 65–72. IEEE.
[Kim 2014] Kim, Y. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
[Kugler 2016] Kugler, L. 2016. What happens when big data blunders?
[Lazer and Kennedy 2015] Lazer, D., and Kennedy, R. 2015. What we can learn from the epic failure of google flu trends. wired, october 2015.
[Lee et al. 2006] Lee, T. E.; Miller, S. D.; Turk, F. J.; Schueler, C.; Julian, R.; Deyo, S.; Dills, P.; and Wang, S. 2006. The npoess viirs day/night visible sensor. Bulletin of the American Meteorological Society 87(2):191–200.
[Levin and Heimowitz 2012] Levin, N., and Heimowitz, A. 2012. Mapping spatial and temporal patterns of mediterranean wildfires from modis. Remote Sensing of Environment 126:12–26.
[Liu, Stanturf, and Goodrick 2010] Liu, Y.; Stanturf, J.; and Goodrick, S. 2010. Trends in global wildfire potential in a changing climate. Forest ecology and management 259(4):685–697.
[Losing, Hammer, and Wersing 2016] Losing, V.; Hammer, B.; and Wersing, H. 2016. Knn classifier with self adjusting memory for heterogeneous concept drift. In 2016 IEEE 16th international conference on data mining (ICDM), 291–300. IEEE.
[Lum and Isaac 2016] Lum, K., and Isaac, W. 2016. To pre- dict and serve? Significance 13(5):14–19.
[Middleton et al. 2018] Middleton, S. E.; Kordopatis-Zilos, G.; Papadopoulos, S.; and Kompatsiaris, Y. 2018. Location extraction from social media: Geoparsing, location disambiguation, and geotagging. ACM Transactions on Information Systems (TOIS) 36(4):40.
[Musaev, Wang, and Pu 2014] Musaev, A.; Wang, D.; and Pu, C. 2014. Litmus: Landslide detection by integrating multiple sources. In ISCRAM.
[Pedruco et al. 2018] Pedruco, P.; Szemis, J.; Brown, R.; Lett, R.; Ladson, A.; Kiem, A.; Chiew, F.; et al. 2018. Assessing climate change impacts on rural flooding in victoria. In Hydrology and Water Resources Symposium (HWRS 2018): Water and Communities, 645. Engineers Australia.
[Popat et al. 2018] Popat, K.; Mukherjee, S.; Str¨otgen, J.; and Weikum, G. 2018. Credeye: A credibility lens for analyzing and explaining misinformation. In Companion Proceedings of the The Web Conference 2018, 155–158. International World Wide Web Conferences Steering Committee.
[Ren et al. 2018] Ren, S.; Liao, B.; Zhu, W.; and Li, K. 2018. Knowledge-maximized ensemble algorithm for different types of concept drift. Information Sciences 430:261– 281.
[Ritter et al. 2012] Ritter, A.; Etzioni, O.; Clark, S.; et al. 2012. Open domain event extraction from twitter. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, 1104–1112. ACM.
[Rong 2014] Rong, X. 2014. word2vec parameter learning explained. arXiv preprint arXiv:1411.2738.
[Sakaki, Okazaki, and Matsuo 2010] Sakaki, T.; Okazaki, M.; and Matsuo, Y. 2010. Earthquake shakes twitter users: real-time event detection by social sensors. In Proceedings of the 19th international conference on World wide web, 851–860. ACM.
[Sakamoto et al. 2015] Sakamoto, Y.; Fukui, K.-I.; Gama, J.; Nicklas, D.; Moriyama, K.; and Numao, M. 2015. Concept drift detection with clustering via statistical change detection methods. In 2015 Seventh International Conference on Knowledge and Systems Engineering (KSE), 37–42. IEEE.
[Scheirer et al. 2012] Scheirer, W. J.; de Rezende Rocha, A.; Sapkota, A.; and Boult, T. E. 2012. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence 35(7):1757–1772.
[Schoennagel et al. 2017] Schoennagel, T.; Balch, J. K.; Brenkert-Smith, H.; Dennison, P. E.; Harvey, B. J.; Krawchuk, M. A.; Mietkiewicz, N.; Morgan, P.; Moritz, M. A.; Rasker, R.; et al. 2017. Adapt to more wildfire in western north american forests as climate changes. Proceedings of the National Academy of Sciences 114(18):4582– 4590.
[Sriram et al. 2010] Sriram, B.; Fuhry, D.; Demir, E.; Fer- hatosmanoglu, H.; and Demirbas, M. 2010. Short text clas-sification in twitter to improve information filtering. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, 841– 842. ACM.
[Stevens-Rumann et al. 2018] Stevens-Rumann, C. S.; Kemp, K. B.; Higuera, P. E.; Harvey, B. J.; Rother, M. T.; Donato, D. C.; Morgan, P.; and Veblen, T. T. 2018. Evidence for declining forest resilience to wildfires under climate change. Ecology Letters 21(2):243–252.
[Suprem and Pu 2019] Suprem, A., and Pu, C. 2019. Assed: A framework for identifying physical events through adaptive social sensor data filtering. In Proceedings of the 13th ACM International Conference on Distributed and Eventbased Systems, 115–126.
[Suprem, Musaev, and Pu 2019] Suprem, A.; Musaev, A.; and Pu, C. 2019. Concept drift adaptive physical event detection for social media streams. In World Congress on Services, 92–105. Springer.
[Suprem 2019] Suprem, A. 2019. Concept drift detection and adaptation with weak supervision on streaming unlabeled data. arXiv preprint arXiv:1910.01064.
[Wang, Hovy, and Dredze 2015] Wang, H.; Hovy, E.; and Dredze, M. 2015. The hurricane sandy twitter corpus. In Workshops at the twenty-ninth AAAI conference on artificial intelligence.