Workshop on Social Events in Web Multimedia, ICMR 2014

1. Social Event Detection at MediaEval: a three-year retrospect of tasks and results Georgios Petkos, Symeon Papadopoulos, Vasileios Mezaris, Raphael Troncy, Philipp Cimiano, Timo Reuter, Yiannis Kompatsiaris Workshop on Social Events in Web Multimedia, ICMR 2014

2. Overview • The problem of social event detection. • The social event detection task. • Evolution of the task and datasets. • Overview of approaches pursued by participants and results. • Outlook.

3. Social events? Attended by people and represented by multimedia content shared online news demonstration / riot / speech personal wedding / birthday / drinks entertainment concert / play / sports

4. Pope Benedict 2007: iPhone release 2008: Android release 2010: iPad release Pope Francis http://petapixel.com/2013/03/14/a-starry-sea-of-cameras-at-the-unveiling-of-pope-francis/

5. Social event detection Social event detection involves the discovery and retrieval of social events in collections of multimedia. COLLECTION EVENT SET E1 E2 SOCIAL EVENT DETECTION EN #5

6. The social event detection task As part of the well-known MediaEval benchmarking activity, a task on social event detection has been running for 3 years (2011-2013). The interest on the task has grown significantly: 2011: 7 participants 2012: 5 participants 2013: 11 participants #6

7. SED 2011: The task Two challenges were defined in 2011. In both, participants were provided with a set of images collected from Flickr and were asked to surface events of a particular type at particular locations: Soccer matches in Barcelona and Rome. Concerts in Paradiso and Parc Del Forum. Two differences between the two challenges: In the first, both a topical and a location criterion are defined, whereas in the second only a location criterion is defined. The specificity of the location of interest is different. #7

8. SED 2011: The dataset, ground truth and evaluation 73,645 photos collected from Flickr. All photos were originally geo-tagged and were taken at 5 different cities in May 2009 (geo-tags were removed for 80% of the pictures in the provided dataset). The ground truth was generated by utilizing machine tags provided by event directories as well as an automatic cluster-based framework. Evaluation measures: F-score NMI #8

9. SED 2012: The task Three challenges, similar to those of the first year, were defined in 2012. Again, participants were provided with a set of images collected from Flickr and were asked to surface events of a particular type at particular locations: Technical events (e.g. exhibitions and fairs) that took place in Germany. Soccer events in Hamburg and Madrid. Demonstration and protest events of the Indignados movement in Madrid. Characteristic of the challenges: Theme and location of queries quite different. Notion of “technical events” somewhat fuzzy. Indignados events are spontaneously organized. #9

10. SED 2012: The dataset, ground truth and evaluation 167,332 photos collected from Flickr. Again, all photos were originally geo-tagged but geo-tags were removed for 80% of the pictures in the provided dataset. The ground truth was again generated by utilizing machine tags provided by event directories as well as an automatic cluster-based framework. Evaluation measures: F-score NMI #10

11. SED 2013: The task Two completely new challenges were defined: Produce a complete clustering of the image dataset according to events. Extension: Assign a set of videos to the clusters generated from the image dataset. Classify event media as either representing a social event or not and for those that do represent a social event identify the type of event (eight event types were defined). #11

12. SED 2013: The dataset, ground truth and evaluation Separate dataset, ground truth and evaluation for each challenge: Challenge 1: Dataset: 427,370 pictures from Flickr and 1,327 videos from YouTube corresponding to 21,169 events. Ground truth: obtained from last.fm and upcoming machine tags. Evaluation: F-score and NMI. Challenge 2: Dataset: 27,754 training images and 29,411 test images collected from Instagram. Ground truth: obtained by manual annotation. Evaluation: NMI. #12

13. Evolution of the task Two distinct eras of the task: First two years. Datasets contained both event and non-event images and the task was to retrieve sets of images matching these criteria. Third year. Broken into two subtasks: Full clustering. Detection of event type in individual images. (no filtering subtask though) Also, datasets have become larger and richer. #13

14. Approaches: First era, 2011-2012 (1/4) At a very high level there are two types of approaches: Matching images to event descriptions retrieved from online event directories. Applying a sequence of filtering or classification and clustering steps on the datasets. #14

15. Approaches: First era, 2011-2012 (2/4) Methods in the first class differ in the way that matching is carried out: Indexing and querying in Lucene. Probabilistic matching. Methods in the second class are much more popular and: Some utilize external sources, e.g. DBPedia or the Google geocoding API to enrich the matching criteria. For most of them time and location (sometimes inferred by the textual metadata when geo-tags are not available) are the primary criteria for clustering. Alternatively, some approaches treat the problem as a multimodal clustering problem utilizing a learned similarity metric #15

16. Approaches: First era, 2011-2012 (3/4) For challenge 1 best approach performs early classification of images into cities and then groups images into buckets containing same day and city photos. For challenge 2, two matching-based approaches achieved the best results (most likely because the type of event makes it more likely to find relevant info in online directories). #16

17. Approaches: First era, 2011-2012 (4/4) The best approach by Vavliakis et al. involves the following steps: City classification. For the images of each city, topic modeling using LDA is performed. The topic model is used to match the photos that are relevant to the queries. Events are identified by finding for each topic and city of interest the days for which there a number of images above some threshold. #17

18. Approaches: Second era, 2013 (1/4) For the first challenge, there are two main types of approaches: Sequence of unimodal clustering operations. Multimodal clustering using a learned similarity measure. However, there are also some rather distinct approaches, e.g.: An approach that applies a Chinese Restaurant Process to perform a stochastic clustering of images. An approach that utilizes WordNet to compute appropriate semantic similarity measures. #18

19. Approaches: Second era, 2013 (2/4) Results are better that in the 2 previous years (probably because a filtering step is not required) The best approach computes one affinity matrix per modality, averages them and uses the average for clustering as part of a DBScan or spectral clustering procedure. #19

20. Approaches: Second era, 2013 (3/4) For the second challenge, all approaches adopt a classification procedure. They differ in the set of features that they utilize. For instance: One approach utilizes scalable Laplacian Eigenmaps to obtain in a semi-supervised manner, an appropriate representation of the images. Another approach used semantic similarity features based on WordNet. #20

21. Approaches: Second era, 2013 (4/4) The best performing approach uses an SVM classifier and a very rich set of textual features, including a set of ontological features (visual features are not used). #21

22. Outlook for the SED task Remarkable number of participants in the last year and appearance of quite novel approaches. The SED task is organized in 2014 as well! Three challenges this year: Full clustering. Retrieval / filtering. Summarization / labelling of events. Registration opens soon! #22

23. Outlook for the problem of social event detection We haven’t seen any approach for dealing with the problem of social event detection “into the wild”: Examined image collections so far had a high ratio of event to non-event photos; the application to a random collection of images would most likely produce poor results. Classification of images as event or non-event related is important for dealing with the more general scenario. Additionally, accurate event/non-event classifiers may assist for obtaining more focused crawling mechanisms. Combination of agnostic approaches (such as clustering) and approaches that utilize event directories. More extensive usage of visual content, rather than mostly of metadata. #23

24. Acknowledgments #24