1 / 21

Mini-Project on Web Data Analysis

Mini-Project on Web Data Analysis . Daniel Deutch. Data Management. “Data management is the development, execution and supervision of plans, policies, programs and practices that control , protect , deliver and enhance the value of data and information assets”

dupdike
Download Presentation

Mini-Project on Web Data Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Mini-Project on WebData Analysis Daniel Deutch

  2. Data Management “Data management is the development, execution and supervision of plans, policies, programs and practices that control, protect, deliver and enhance the value of data and information assets” (DAMA Data Management Body of Knowledge) A major success: the relational model of databases

  3. Relational Databases • Developed by Codd (1970), who won the Turing award for the model • Huge success and impact: • The vast majority of organizational datatoday is stored in relational databases • Implementations include MS SQL Server, MS excel, Oracle DB, mySQL,… • 2 Turing award winners (Edgar F. Codd and Jim Gray) • Basic idea: data is organized in tables (=relations) • Relations can be derived from other relations using a set of operations calledthe relational algebra • On which SQL is largely based

  4. Research in Data(base) Management • 1970- : Relational Databases (tables). • Indexing, Tuning, Query Languages, Optimizations, Expressive Power,…. • ~20 years ago: Emergence of the Web and research on Web data • XML, text database, web graph…. • Google is a product of this research (by Stanford’s PhD students Brin and Page) • Recent years: hot topics include distributed databases, data privacy, data integration, social networks, web applications, crowdsourcing,trust,… • Foundations taken from “classical” database research • Theoretical foundations with practical impact

  5. Web 2.0 • “Old” web (“Web 1.0”): static pages • News, encyclopedic knowledge... • No, or very little, interactive process between the web-page and the user. • Web 2.0: A term very broadly used for web-sites that use new technologies (Ajax, JS..), allowing interaction with the user. • “Network as platform" computing • The “participatory Web”

  6. Web 2.0 • “Old” web (“Web 1.0”): static pages • News, encyclopedic knowledge... • No, or very little, interactive process between the web-page and the user. • Web 2.0: A term very broadly used for web-sites that use new technologies (Ajax, JS..), allowing interaction with the user. • “Network as platform" computing • The “participatory Web”

  7. Online shopping

  8. Advertisements

  9. Social Networks

  10. Crowd Sourcing

  11. Data is all around • Web graph • “Social graph” • Pictures, Videos, notifications, messages.. • Data that the application processes • Advertisments • Even the application structure itself

  12. (A small portion of) the web graph

  13. Need to Analyze • Huge amount of data out there • Est. 13.68 billion web-pages and counting • Half a billion tweets per day and counting • An average user “sees” about 600 tweets per day • Most of it is irrelevant for you, some is incorrect

  14. Filter, Rank, Explain • Filter • Select the portion of data that is relevant • Group similar results • Rank • Rank data by trustworthiness, relevance, recency... • Present highest-rank first • Explain • An explanation of whyis the data considered relevant/highly-ranked • An explanation of howhas the data propagated • “Why do I see this?”

  15. Main topics • Analysis of Tables and Links on the Web • Trust Management • Explanation (Provenance) • Information Extraction • Social Networks • Crowd-sourcing • Distributed Query Evaluation

  16. Approach • Leverage knowledge from “classic” database research • Account for the new challenges • Do so in a generic manner • Leverage unique features such as collaborative contribution, distribution, etc.

  17. sname sid=sid cid=cid name=“Mary” Courses Takes Students Data model Query language Students Select… From… Where… Physical Storage Indexing Distribution ...

  18. Foundations • Model • Query Language • Query evaluation algorithms • Prototype implementation and optimizations • Getting Data and Testing

  19. Project Requirements • Read a paper (or a bunch of papers) in the area • Likely to require that you follow citations and read earlier papers! • Think of an application based on the paper ideas • Does not have to be exactly the application described in the paper! • E.g. you do not have to use relational databases • Think of how would you get/generate data • Implement, test • Submit an application+ report

  20. Report • An integral part of the project submission • Should include: • A detailed description of the model and algorithms that you have implemented • A detailed description of the application • Code design • Use cases • Difficulties that you have encountered and how you addressed them

  21. Timeline • By 20/3 (1 week from now): send me an ordered list of 3 preferred papers • Email title includes the words “mini-project” • Body includes the names and IDs of the pair • A bit after passover (date TBA): Each pair presents a 7-10 minutes presentation on the expected project A slide on each of the issues mentioned in the requirement slide 1 week before the last week of the semester: short project presentations (including screenshots or live demo)

More Related