1 / 18

Pre-Workshop Deck

Pre-Workshop Deck. Map Reduce Workshop Monday November 12 th , 2012. Overview: Schedule. 5:30-6:00  Networking, Software Install, Cloud Setup 6:00-6:10  M/R and Workshop Overview - John Verostek 6:10-7:20  Map/Reduce Tutorial - Vipin Sachdeva (IBM Research Labs)

chaim
Download Presentation

Pre-Workshop Deck

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pre-Workshop Deck Map Reduce Workshop Monday November 12th, 2012

  2. Overview: Schedule 5:30-6:00  Networking, Software Install, Cloud Setup 6:00-6:10  M/R and Workshop Overview - John Verostek 6:10-7:20  Map/Reduce Tutorial - VipinSachdeva (IBM Research Labs) The Map/Reduce Programming Framework will be introduced using a hands-on Word Count example using Python.  Next the basics of Hadoop Map/Reduce and File Server will be covered.  Time permitting, a demo will be given of running the Python M/R program using Hadoop installed locally. 7:20-7:30 Short Break 7:30-8:45  Applications using Amazon Elastic M/R - J Singh (DataThinks.org)  A Facebook application will also be walked through.  For this dataset, everyone who attends the workshop will have the option to sign into a workshop prep page with their Facebook account and give permission to share their likes.  The data is automatically anonymized and sent to an Amazon S3 file.  The exercise will find likes common to people in the sample. What might someone do after the analysis of such data? Design an advertising campaign, perhaps (but designing an ad campaign is not part of the workshop).

  3. Pre-Workshop Checklist • Cloud: Amazon Web Services to be used • Facebook Likes Exercise: • App used to anonymously collect “Likes” • Dataset to be used for M/R exercise • Software Installation: • Python will be used to run programs locally • Download 2.7.3, set Environmental Variables • Code and Datasets: Included are links to Files located up on Amazon • Running Python Locally: • Various screen shots to try out a regular (not a Map / Reduce) wordcount program

  4. Cloud Account Please sign-up for an account here: http://aws.amazon.com/ Amazon’s Elastic Map/Reduce will be used. The following 6-minute AWS video shows a wordcount example that is somewhat similar to what will be used in the workshop. There is enough info in this video to within 15-minutes run a M/R job. http://www.youtube.com/watch?v=kNsS9aDf6uE

  5. We will be using Elastic MapReduce and S3

  6. Python Scripts and Wikipedia Datasets sorter.py for folks with Windows machines After loading the link into the browser, and text appearing, then use “right-click”, “save as” These five files of different sizes were created by Vipinto test out the time to run each one locally. Please note the 8GB may take a while to download.

  7. Facebook Likes Exercise Please sign in into a workshop prep page with your Facebook account and give permission to share their likes. If you don't have a Facebook account then no worries. If everyone opts out, we won't have much data to work with. All collected data will be anonymized and then deleted after the workshop is done. The exercise will find likes common to people in the sample. What might someone do after the analysis of such data? Design an advertising campaign, perhaps (though designing an ad campaign is not part of the workshop). • The App that collects Facebook data is: • http://apps.facebook.com/map-reduce-workshop

  8. Should be where you can just copy and paste the URL into the browser where you have Facebook set up Then something like this should appear after it has pulled over the Likes.

  9. Python Download • Mac/Linux comes installed with Python (should be able to run). • Windows : if you do not already have Python installed, then use the following website to download and install: • http://www.python.org/download/ • DOWNLOAD VERSION 2.7.3 • DO NOT DOWNLOAD VERSION 3.3.0

  10. Python Installation

  11. Python Environmental Variables There are many online instructions that explain this such as the below link: http://pythoncentral.org/how-to-install-python-2-7-on-windows-7-python-is-not-recognized-as-an-internal-or-external-command/

  12. More Python Help http://docs.python.org/2/using/

  13. We will be using the Command Line • Get to the DIR where your code/data is located • CD to call a directory • CD.. to go up a level • e.g. cd users\john\desktop\python

  14. Wordcount - Very Simple Python Script – seq.py #!/usr/bin/env python import sys import re counts={} for line in sys.stdin: words = re.split("\W+",line) for word in words: counts[word]=counts.get(word,0)+1; for x,y in counts.items(): print x,y Import system functions and regular expressions library Read from stdin Split along newline character (W+) Let’s try running it!! Use Wikipedia Dataset filename Output File – Can be any name Python Code seq.py > python ./seq.py < input.txt > output.txt Make sure you have the < and > pointing in the right directions or could over-write input file.

  15. No Output File so goes to screen > python ./seq.py < input.txt CTRL-C to break

  16. Time Mac OS/Linux/Cygwin • > time python ./seq.py < input.txt Windows If you run the datasets locally, and “time” how long they take (output to file not screen), then then please email me the results (including laptop info, etc) as we are making a composite slide of “time vs. file size.” • * I noted the start time, 11:46 PM and then ran overnight and went with the timestamp on the output file; 03:54 AM.

More Related