1 / 34

ACM KDD Cup A Survey: 1997-2011

ACM KDD Cup A Survey: 1997-2011. Qiang Yang 杨强 (partly based on Xinyue Liu’s slides @SFU, and Nathan Liu’s slides @hkust) Hong Kong University of Science and Technology 香港科大. About KDD Cup (1997 – 2011). C ompetition is a strong mover for Science and Engineering: ACM Programming Contest

deanna
Download Presentation

ACM KDD Cup A Survey: 1997-2011

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ACM KDD Cup A Survey: 1997-2011 Qiang Yang 杨强 (partly based on Xinyue Liu’s slides @SFU, and Nathan Liu’s slides @hkust) Hong Kong University of Science and Technology 香港科大

  2. About KDD Cup (1997 – 2011) • Competition is a strong mover for Science and Engineering: • ACM Programming Contest • World College level Programming skills • ROBOCUP • World Robotics Competition

  3. About ACM KDDCUP • ACM KDD: Premiere Conference in knowledge discovery and data mining • ACM KDDCUP: • Worldwide competition in conjunction with ACM KDD conferences. • It aims at: • showcase the best methods for discovering higher-level knowledge from data. • Helping to close the gap between research and industry • Stimulating further KDD research and development

  4. Statistics • Participation in KDD Cup grew steadily • Average person-hours per submission: 204Max person-hours per submission: 910

  5. Algorithms (up to 2000)

  6. KDD Cup 97 • A classification task – to predict financial services industry (direct mail response) • Winners • Charles Elkan, a Prof from UC-San Diego with his Boosted Naive Bayesian (BNB) • Silicon Graphics, Inc with their software MineSet • Urban Science Applications, Inc. with their software gain, Direct Marketing Selection System

  7. MineSet (Silicon Graphics Inc.) • A KDD tool that combines data access, transformation, classification, and visualization.

  8. KDD Cup 98: CRM Benchmark • URL:www.kdnuggets.com/meetings/kdd98/kdd-cup-98.html • A classification task – to analyze fund raising mail responses to a non-profit organization • Winners • Urban Science Applications, Inc. with their software GainSmarts. • SAS Institute, Inc. with their software SAS Enterprise Miner ™ • Quadstone Limited with their software Decisionhouse ™

  9. KDDCUP 1998 Results Maximum Possible Profit Line ($72,776 in profits with 4,873 mailed) Mail to Everyone Solution ($10,560 in profits with 96,367 mailed) GainSmarts SAS/Enterprise Miner Quadstone/Decisionhouse

  10. ACM KDD Cup 1999 • URL: www.cse.ucsd.edu/users/elkan/kdresults.html • Problem To detect network intrusion and protect a computer network from unauthorized users, including perhaps insiders • Data: from DoD • Winners • SAS Institute Inc. with their software Enterprise Miner. • Amdocs with their Information Analysis Environment

  11. Data collected from Gazelle.com, a legwear and legcare Web retailer Pre-processed Training set: 2 months Test sets: one month Data collected includes: Click streams Order information The goal – to design models to support web-site personalization and to improve the profitability of the site by increasing customer response. Questions - Whengiven a set of page views, characterize heavy spenders characterize killer pages characterize which product brand a visitor will view in the remainder of the session? KDDCUP 2000: Data Set and Goal:

  12. KDDCUP 2000: The Winners • Question 1 & 5 Winner: Amdocs • Question 2 & 3 Winner: Salford Systems • Question 4 Winner: e-steam

  13. 3 Bioinformatics Tasks Dataset 1: Prediction of Molecular Bioactivity for Drug Design half a gigabyte when uncompressed Dataset 2: Prediction of Gene/Protein Function (task 2) and Localization (task 3) Dataset 2 is smaller and easier to understand 7 megabytes uncompressed A total of 136 groups participated to produce a total of 200 submitted predictions over the 3 tasks: 114 for Thrombin, 41 for Function, and 45 for Localization. KDD Cup 2001

  14. Task 1, Thrombin: Jie Cheng (Canadian Imperial Bank of Commerce). Bayesian network learner and classifier Task 2, Function: Mark-A. Krogel (University of Magdeburg). Inductive Logic programming Task 3, Localization: Hisashi Hayashi, Jun Sese, and Shinichi Morishita (University of Tokyo). K nearest neighbor Task 2: the genes of one particular type of organism A gene/protein can have more than one function, but only one localization. 2001 Winners

  15. molecular biology : Two tasks Task 1: Document extraction from biological articles Task 2: Classification of proteins based on gene deletion experiments Winners: Task 1: ClearForest and Celera, USA Yizhar Regev and Michal Finkelstein Task 2: Telstra Research Laboratories, Australia Adam Kowalczyk and Bhavani Raskutti

  16. 2003 KDDCUP • Information Retrieval/Citation Mining of Scientific research papers • based on a very large archive of research papers • First Task: predict how many citations each paper will receive during the three months leading up to the KDD 2003 conference • Second Task: a citation graph of a large subset of the archive from only the LaTex sources • Third Task: each paper's popularity will be estimated based on partial download logs • Last Task: devise their own questions

  17. 2003 KDDCUP: Results • Task 1: • Claudia Perlich, Foster Provost, Sofus Kacskassy • New York University • Task 2: • 1st place: David Vogel • AI Insight Inc. • Task 3: • Janez Brank and Jure Leskovec • Jozef Stefan Institute, Slovenija • Task 4: • Amy McGovern, Lisa Friedland, Michael Hay, • Brian Gallagher, Andrew Fast, Jennifer Neville, • and David Jensen • University of Massachusetts • Amherst, USA

  18. 2004 Tasks and Results • 粒子物理学和同调蛋白质预测（Particle physics; plus protein homology prediction） • 两个子任务的冠军分别为：David S. Vogel, Eric Gottschalk, and Morgan C. Wang以及Bernhard Pfahringer, Yan Fu (付岩), RuiXiang Sun, Qiang Yang (杨强), Simin He, Chunli Wang, Haipeng Wang, Shiguang Shan, Junfa Liu, Wen Gao.

  19. Past KDDCUP Overview: 2005-2010

  20. KDDCUP’11 Dataset • 11 years of data • Rated items are • Tracks • Albums • Artists • Genres • Items arranges in a taxonomy • Two tasks

  21. Items in a Taxonomy

  22. Track 1 Details

  23. Track 1 Highlights • Largest publicly available dataset • Large number of items (50 times more than Netflix) • Extreme rating sparsity (20 times more sparse than Netflix) • Taxonomy can help in combating sparsely rated items. • Fine time stamps with both date and time allow sophisticated temporal modeling.

  24. Track 2 Details

  25. Track 2 Highlights • Performance metric focus on ranking/ classification, which differs from traditional collaborative filtering. • No validation data provided, need to self-construct binary labeled data from rating data. • Unlike track 1, track 2 removed time stamps to focus more than long term preference rather than short term behaviors.

  26. Submission Stats

  27. Winners

  28. Chinese Teams at KDDCUP (NTU, CAS, HKUST)

  29. Key Techniques • Track 1: • Blending of multiple techniques • Matrix factorization models • Nearest neighbor models • Restricted Bolzmann machines • Temporal modelings • Track 2: • Importance sampling of negative instances • Taxonomical modelings • Use of pairwise ranking objective functions

  30. Summary • To place on top of KDDCUP requires • Team work • Expertise in domain knowledge as well as mathematical tools • Often done by world famous institutes and companies • Recent trends: • Dataset increasingly more realistic • Participants increasingly more professional • Tasks are increasingly more difficult

  31. Summary • KDD Cup is an excellent source to learn the state-of-art KDD techniques • KDDCUP dataset often becomes the standard benchmark for future research, development and teaching • Top winners are highly regarded and respected

  32. References Elkan C. (1997). Boosting and Naive Bayesian Learning. Technical Report No. CS97-557, September 1997, UCSD. Decisionhouse (1998). KDD Cup 98: Quadstone Take Bronze Miner Award. Retrieved March 15, 2001 fromhttp://www.kdnuggets.com/meetings/kdd98/quadstone/index.html Urbane Science (1998). Urbane Science wins the KDD-98 Cup. Retrieved March 15, 2001 from http://www.kdnuggets.com/meetings/kdd98/gain-kddcup98-release.html Georges, J. & Milley, A. (1999). KDD’99 Competition: Knowledge Discovery Contest. Retrieved March 15, 2001 from http://www.cse.ucsd.edu/users/elkan/saskdd99.pdf Rosset, S. & Inger A. (1999). KDD-Cup 99 : Knowledge Discovery In a Charitable Organization’s Donor Database. Retrieved March 15, 2001 from http://www.cse.ucsd.edu/users/elkan/KDD2.doc

  33. References (Cont.) Sebastiani P., Ramoni M. & Crea A. (1999). Profiling your Customers using Bayesian Networks. Retrieved March 15, 2001 from http://bayesware.com/resources/tutorials/kddcup99/kddcup99.pdf Inger A., Vatnik N., Rosset S. & Neumann E. (2000). KDD-Cup 2000: Question 1 Winner’s Report. Retrieved March 18, 2000 from http://www.ecn.purdue.edu/KDDCUP/amdocs-slides-1.ppt Neumann E., Vatnik N., Rosset S., Duenias M., Sasson I. & Inger A. (2000). KDD-Cup 2000: Question 5 Winner’s Report. Retrieved March 18, 2000 from http://www.ecn.purdue.edu/KDDCUP/amdocs-slides-5.ppt Salford System white papers: http://www.salford-systems.com/whitepaper.html Summary talk presented at KDD (2000) http://robotics.stanford.edu/~ronnyk/kddCupTalk.ppt

  34. References (cont) • http://www.cs.wisc.edu/~dpage/kddcup2001/Cheng.pdf • http://www.cs.wisc.edu/~dpage/kddcup2001/Krogel.pdf • http://www.cs.wisc.edu/~dpage/kddcup2001/Hayashi.pdf

More Related