1 / 35

SnatZ Technology

SnatZ Technology. as news analysis tool. Main terms used in presentation. Term – a phrase, which system uses for training NLP algorithms. Summary – a phrase, which system automatically detects during analyzing of news content.

talia
Download Presentation

SnatZ Technology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SnatZTechnology as news analysis tool

  2. Main terms used in presentation Term – a phrase, which system uses for training NLP algorithms. Summary – a phrase, which system automatically detects during analyzing of news content. Trend - an unique chain, which contains one, two or more summary. These chains are created as result of cluster analysis. Tag– a term, which created by moderator for detecting user’s interest category. User interests - cloud of tags, which system recognizes from user’s social accounts and OPML files. Segments - a groups of ‘similar’ Trends, which are intersected more than 30% by search results. Semantic network - is a network which represents semantic relations between keywords Data warehouse - is a database used for reporting and data analysis.

  3. The main goal of SNATZ • Snatz is a data mining instrument. • It can recognizes semantic of news content using NLP algorithm • On the basis of acquired summary SNATZ candefinenew knowledges: • detect new Summary sets • gathering Trends statistics • opportunity to build Segments • usingnew Summary as Terms for training NLP algorithm • Making recommendation of news from different Segments • Our solutions allow to change the paradigm of ‘Collaborative Filtering’

  4. Snatz platform architecture • Snatz platform architecture consists of: • SNATZ Recommender System - personal recommendation based on the users’ interests • SNATZ Data Mining Tool – semantic network of trends. It is created by sending recognized metadata to analysis processing

  5. SNATZ Recommender • CRAWLER. Crawler interacts with Web sites by receiving RSS-feeds and tweets. Content of RSS and tweets are the main resources. • Blogosphere. All resources which web crawler detected are saved in data warehouse makes internal SNATZ “blogosphere”. • Data Processing. Exporting resources to the SNATZ DM Tool. Also, together with news articles it sends sets of labels, terms, summary. • USERS • Tags/Posts: • - Posts. System recognized users’ posts from Fb, Tw and OPML files. • - Tags. Using NLP algorithm system defines the User’s interests. • Recommendations. Component contains rules of forming news recommendations. • News archive. News items which were recommended for a users

  6. SNATZ Data Mining Tool • Documenter. Imports resources from Data Processing component and sends them to the NLP • NLP. Semantic analysis: • - POS Tagging • - Defining articles attributes: labels, terms and summary. • Meta-Docs. Data warehouse of articles with semantic analysis • Analysis: • - Multi Clusterization • - Trends defining • Semantic Network of Trends: • - Segments • Reporter • - Trends statistics

  7. Data workflow Data Mining Tool Recommender System Data Processing Docs IMPORT WEB NLP EXPORT Blogs CRAWLER News Archive Meta Docs Analysis Users Semantic Network of Trends Tags/ Posts Reporter Recommendation

  8. Building segments Meta-Docs Documents Tree of Trends • Meta –Docs • Labels • Terms • Summary Tree of Trends Segments

  9. Building segments • Documenter imports resources and sets of labels, terms, summary from Data Processing component. And sends them to the NLP. • NLP recognizes an attributes in recourses: labels, terms, summary. These resources become a meta-docs and are saved in Data warehouse. • Meta Docs are sent to the Analysis and system forms actual Trend Tree. • Trends identify related summary, i.e. the main direction of its topics and sub-topics. Through such relations of trends SNATZ finds similar/related topics and groups them into Segments: • - If Trends intersects more than 30% than trends create a new Segment.

  10. Recommendations Meta-Docs Users Posts Interests Tags Related Tags Daily Review

  11. Recommendations • System parses users posts from Fb, Tw, uploaded OPML file of subscriptions. • Updating of interests performed every 4 hours • NLP recognized interests from Posts resulting a set of Tags. • If number of Tags is less than 12, system tries to find relates Tags. • System takes Trends which were received from Meta-Docs and defines related Trends for users interests. If Trend contains user’s interest it becomes connected with user. • Summary which are in related Trend becomes the Related Tags. • System takes trends from User’s Trend tree and makes Daily Review

  12. Personal Recommendations 'Diversity' Filtering Get Trends Check Trends Interests User Interests Segments User’s Trends User’s Trends List of 12 News User’s Tree of Trends

  13. Personal Recommendations • System takes ‘Last Trends’ which contains users interests and forms User’s Trends • User’s Trends are checked on segments and forms User’s Trends Tree. • ’Diversity’ filtering: • System does not take more than 2 interests from one category • No more than one news article for the trend • System gets news only with new keywords (i.e. comparing with previous sets of news) • Only 1 news from same segment • Only 2 news from one category

  14. SNATZ server architecture

  15. SNATZ server architecture • Cluster High-availability provides the following services: • virtual ip for cluster. • DRBD storage of cluster . • ext4 file system on top of DRBD. • containers openVZ on ext4 over DRBD. • each cluster is assembled on two nodes. • corosyn is used for managing. • Pacemaker is a resource manager. • system is five two-node clusters.

  16. SNATZ server architecture • Redundant services are performed on openVZ containers and start • together with the start of the container. • Interaction redundant services between the containers is carried via the local network, which is connected via a separate commutator to the second network interface of each node. • For each two-node cluster written sequence of start of redundant services: • 1. switching active / passive DRBD • 2. mount the ext4 file system to the mount point of the active node . • 3. start of openVZ containers which are placed on DRBD.

  17. SNATZ + Elasticsearch engine • Elasticsearch is a search server which provides distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. • Advantages Elasticsearch for SNATZ: • Elasticsearch is a stable working project • AWS Cloud Plugin (allows to use Amazon EC2 API) • Real time data Search and Analysis • Index versioning support • Search opportunities: fuzzy requests & etc.

  18. Elasticsearch + Amazon EC2

  19. Elasticsearch + Amazon EC2 Features: • ability to maintain a high performance cluster designed for I/O intensive operations • new instances are started and stopped when required • no need to pay for long-term servers and their administration • pricing is per instance-hour consumed for each instance • ability to create images from a working machine (configured & set up) and start other instances from these images

  20. SNATZ + NLP • Features: • Part-of-speech-tagging • Summary extraction • User-defined Terms and Labels • Synonyms handling • Supervised text classification using user-defined datasets for training/evaluating performance • Language support: • English • Japanese (using third-party tools like MeCab)

  21. Challenges of SNATZ • Filter Bubble (user’s interests) • Diversity and ‘Long Tail’ • Data sparsity (‘the cold start problem’) • Scalability • Segmentation (‘related topics’)

  22. How SNATZ solves this problems? Using TRENDs

  23. What is Filter Bubble User can see popular news only by TOP-Tags from his interests’ categories. But user doesn’t see related Tags outside the Filter Bubble

  24. What is TREND? All summary and terms of articles has close connections. The task of SNATZ to define significant connections.

  25. How Trends are detected? News with Terms Clustering By Terms Clustering By Summary System detected Trends Clustering By Labels

  26. Abstraction of algorithm • Multilevel clustering algorithm has 3 abstractions: • Labels • Terms • Summary

  27. SNATZ outside Filter bubble SNATZ tries to show news beyond users` filter bubble to cover more Trends. Trends identify related summary, i.e. the main direction of its topics and sub-topics

  28. Long Tail problem • Users usually doesn't see most of news because they have too small Popularity Rank. • SNATZ solves this problem: • For user recommendations SNATZ selects Trends only by different Segments • In order to provide users with *new* content, SNATZ does NOT make recommendations based on Summary that were already picked for previous recommendations. This way the user can see the news based on the latest Trends • SNATZ does NOT use TOP-Tags from user’s interest categories.

  29. Collaborating Filter The first and most common way to determine the significance of an article is its social rating. This is determined through an advanced technique called Collaborative Filtering, which collects taste preferences or personal information (such as language, country, etc.) from many users and uses that data to make automatic predictions. SNATZ recommends news solely on the basis of user interests. Every step of recommendations is unique and depends on the previous step. Recommendations are made only on the basis of the individualuser's experience.

  30. Effective Content Personalization • One approach to effective content personalization is called ‘the classification of trends’, and based on the principles of identifying the most significant relationships between summary, creating a unique chain of summary called a trend. • A trend contains one or more summary from Web content, and determines specific subtopics. The main characteristic of a trend is dynamics of chains or summary, with positive (growing) or negative (fading) conditions over a specific period of time.

  31. Automaticsegmentation of blogosphere Through such relations of chains SNATZ finds similar/related topics and groups them into Segments. For example: ‘Network+Tumblr’ intersects with ‘Network+Tumblr+Instagram’ by more than 30%. These chains create a new Segment. Trends are determined by analyzing content in the current news state of the daily Blogosphere at it’s most basic form - relevant daily news topics. If a recommendation engine calculates the thematic proximity of trends, then it can auto-classify them into trend segments, so that similar sub-topics are put in the same segments. This auto-classification of segments splits Web content on various major topics.

  32. Automaticsegmentation • A recommendation engine that applies this classification process on trends (and not tags) solves two major personalization problems: • Removes Long Tail, making news recommendations from different segments possible • Solves the problem of thematic proximity, making sure that similar or duplicate news is filtered out

  33. Data mining result. Infographics • System can: • detects more actual Trends • for any given topic. • detects ‘Related’ Tags • for any given topic. • detects the dynamics of Trends • detects the sentiment of news System detect

  34. Findings Information becomes increasingly dense, consumers deserve to get the news that they want to read – not the news an algorithm thinks they want. SNATZ gives is a personalization algorithm that can solve the challenges of the filter bubble and long tail

  35. Thanks for your attention! SNATZ Team

More Related