1 / 14

ChaLearn HEP competitions

ChaLearn is a California non-profit organization that organizes machine learning and AI challenges to solve high-impact problems. With a focus on data, clear objectives, and good metrics, ChaLearn encourages participation through exciting problem descriptions, rewards, and open-source code.

fgriffin
Download Presentation

ChaLearn HEP competitions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ChaLearnHEP competitions for Isabelle Guyon, ChaLearn and LRI, Université Paris Sud, Orsay http://chalearn.org

  2. What is Chalearn? • California non-profit (US tax exempt). • Organizes challenges since 2005. • Funded by government grants and industry sponsors. • 10 directors; 4 officers; dozens of collaborators. • CiML book series (JMLR reprints) and NIPS workshops. • Now 3 to 4 challenges per year in 4 main tracks: • Automatic Machine Learning. • Computer vision: Looking At People (LAP) challenges. • Causality and network reconstruction. • High Energy Physics. http://chalearn.org

  3. Recent challenges 2010-15: LAP Challenges (on Kaggle and Codalab) 2013: Neural Connectomics (on Kaggle) 2014: Higgs Boson (on Kaggle) http://chalearn.org

  4. Code Result submission 2010-15: LAP Challenges (on Kaggle and Codalab) Videos: hours of footage. 2013: Neural Connectomics (on Kaggle) Towards billions of connections. 2014: Higgs Boson (on Kaggle) 40 events per sec x 1 year. http://chalearn.org

  5. Code submission on Codalab BIG DATA Tasks Code Scores User Results http://chalearn.org

  6. Testing code submission AutoML Challenge Fully automatic machine learning without ANY human intervention >500 participants automl.chalearn.org December 2014 – January 2016 $30,000 in prizes

  7. Challenge design http://chalearn.org

  8. 1. Define • High impact problem. • Good data. • Enough data. • Clear objective. • Good metric. • Simple rules. • No IP contingencies (except for winners). http://chalearn.org

  9. 2. Implement • Three-way data split (train/valid/test): • Immediate feed-back on validation set. • Final results on test set. • Encourage sharing and novelty (coopetition): • Forum. • Worksheets or scripts. • Monitor downloads / likes / thanks. • Type of submission: • Result. • Code. • Beta testing. http://chalearn.org

  10. 3. Attract • Exciting problem description. • Tutorial. • Video. • Starting kit. • Bootcamp/hackathon. • Publication outlets. • Advertising. • Gamification. http://chalearn.org

  11. 4. Reward • Multiple prizes (phases and/or tracks). • Medium size prize pool (~ $5000-$10,000). • Travel awards. • Best paper awards. • Give credit and rewards to all actors. http://chalearn.org

  12. 5. Harvest • Open-source code (winners). • Fact sheets. • Post-challenge analyses. • Workshops (broader scope than just the challenge theme). • Proceedings. • Crowd-sourced paper. http://chalearn.org

  13. Build/maintain/expand Create recurring events http://chalearn.org

  14. Thank you! Directors Kristin Bennett, Rensselaer Polytechnic Institute, New-York, USA Gideon Dror, Academic college of Tel-Aviv Yaffo, Israel Hugo-Jair Escalante Balderas, Nat. Institute of Astrophysics, Optics and Electronics (INAOE), Mexico Sergio Escalera, University of Barcelona, Spain Chih-Jen Lin, National Taiwan University, Taiwan Florin Popescu, Fraunhofer Institute, Berlin, Germany Mehreen Saeed, FAST, National University of Computer and Emerging Sciences, Lahore Campus, Pakistan Danny Silver, Acadia University, Canada Alexander Statnikov, New York University, New-York, USA Ioannis Tsamardinos, University of Crete, Greece Officers President: Isabelle Guyon, ClopiNet, California, USA Vice-president: Gideon Dror, Academic college of Tel-Aviv Yaffo, Israel Secretary: Vincent Lemaire, Orange Labs, France Treasurer: Kristin Bennett, RPI, USA Sponsors http://automl.chalearn.org

More Related