Machine Learning in realtime

1. Machine Learning in realtime Hilary Mason h@bit.ly @hmason

3. What’s a bitly?

4. http://www.pcworld.com/article/223409/move_over_dr_soong_ girls_can_build_android_apps_too.html http://bit.ly/hOnbWg

8. Email IM Twitter mobile Facebook Google+ bitly!

9. Our challenge: What’s happening on the internet in realtime.

10. Our analysis:analyticsproductscience

11. Our challenge: What’s happening on the internet in realtime.

12. [zoom in]

13. What spoken languages are in a page?

14. raw data "es" "en-us,en;q=0.5" "pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4" "en-gb,en;q=0.5" "en-US,en;q=0.5" "es-es,es;q=0.8,en-us;q=0.5,en;q=0.3” "de, en-gb;q=0.9, en;q=0.8"

16. entropy calculation def ghash2lang(g, Ri, min_count=3, max_entropy=0.2): """ returns the majority vote of a langauge for a given hash """ lang = R.zrevrange(g,0,0)[0] # let's calculate the entropy! # possible languages x = R.zrange(g,0,-1) # distribution over those languages p = np.array([R.zscore(g,langi) for langi in x]) p /= p.sum() # info content I = [pi*np.log(pi) for pi in p] # entropy: smaller the more certain we are! - i.e. the lower our surprise H = -sum(I)/len(I) #in nats! # note that this will give a perfect zero for a single count in one language # or for 5K counts in one language. So we also need the count.. count = R.zscore(g,lang) if count < min_count and H > max_entropy: return lang, count else: return None, 1

18. http://4sq.com/96kc1O

19. How do people click on URLs over time?

21. normal click distributions

26. abnormal click distributions

30. clusted URLs

31. What is the half life of a link?

37. Does this apply to other dimensions?

43. Can we predict at time t how many clicks a link is likely to ever get?

45. Text analysis?

47. How do we rank document relevance in realtime?

50. Simple> Complex (especially algorithms)