1 / 20

MapReduce, Dictionaries, List Comprehensions

Learn about MapReduce, Dictionaries, and List Comprehensions in Python for efficient data processing. Explore examples and use cases for these powerful tools.

rubie
Download Presentation

MapReduce, Dictionaries, List Comprehensions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. MapReduce, Dictionaries, List Comprehensions Special thanks to Scott Shawcroft, Ryan Tucker, and Paul Beck for their work on these slides. Except where otherwise noted, this work is licensed under:http://creativecommons.org/licenses/by-nc-sa/3.0

  2. Map • map(function, iterable, ...) • Map applies function to each element of iterable and creates a list of the results • You can optionally provide more iterables as parameters to map and it will place tuples in the result list • Map returns an iterator which can be cast to list

  3. Map Examples

  4. Recursion The same as in Java!

  5. Functions as Parameters Functions can be assigned to variables and/or passed as parameters

  6. Map Code

  7. Reduce • reduce(function, iterable[,initializer]) • Reduce will apply function to each element in iterable along with the sum so far and create a cumulative sum of the results • function must take two parameters • If initializer is provided, initializer will stand as the first argument in the sum • Unfortunately in python 3 reduce() requires an import statement • from functools import reduce

  8. Reduce Examples

  9. Reduce Examples

  10. Reduce Problem Goal: given a list of numbers I want to find the average of those numbers in a few lines using reduce() For Loop Method: - sum up every element of the list - divide the sum by the length of the list

  11. Reduce Problem

  12. MapReduce Framework for processing huge datasets on certain kinds of distributable problems Map Step: - master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. - worker node may chop its work into yet small pieces and redistribute again

  13. MapReduce Reduce Step: - master node then takes the answers to all the sub-problems and combines them in a way to get the output

  14. MapReduce Problem: Given an email how do you tell if it is spam? - Count occurrences of certain words. If they occur too frequently the email is spam.

  15. MapReduce

  16. Dictionaries map keys, which can be any immutable type, to values, which can be any type

  17. Dictionaries Operations: - print, del, len, in Methods: - keys(), values(), items()

  18. List Comprehensions • [expression for elementin list] • Applies the expression to each element in the list • You can have 0 or more for or if statements • If the expression evaluates to a tuple it must be in parenthesis

  19. List Comprehensions

  20. List Comprehensions You can do most things that you can do with map, filter and reduce more nicely with list comprehensions The email spam program from earlier using list comprehensions:

More Related