1 / 6

厦门大学数据库实验室 MapReduce 排序

厦门大学数据库实验室 MapReduce 排序. 报告人：李雨倩 导师：林子雨 2014.07.19. 在如下示例中，用户数据中有用户姓名，年龄和所在州

kylynn-wall
Download Presentation

厦门大学数据库实验室 MapReduce 排序

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 厦门大学数据库实验室 MapReduce 排序 报告人：李雨倩 导师：林子雨 2014.07.19

  2. 在如下示例中，用户数据中有用户姓名，年龄和所在州在如下示例中，用户数据中有用户姓名，年龄和所在州 • $ cat test-data/ch4/users.txtanne 22 NYjoe 39 COalison 35 NYmike 69 VAmarie 27 ORjim 21 ORbob 71 CAmary 53 NYdave 36 VAdude 50 CA 用户活动日志中有用户姓名，进行的动作， 来源IP。这个文件一般都要比用户数据要大得多。 • $ cat test-data/ch4/user-logs.txtjim logout 93.24.237.12mike new_tweet 87.124.79.252bob new_tweet 58.133.120.100mike logout 55.237.104.36jim new_tweet 93.24.237.12marie view_user 122.158.130.90 $ hadoop fs -put test-data/ch4/user-logs.txt user-logs.txt $ bin/run.sh com.manning.hip.ch4.joins.improved.SampleMain users.txt,user-logs.txt output $ hadoop fs -cat output/part*bob 71 CA new_tweet 58.133.120.100jim 21 OR logout 93.24.237.12jim 21 OR new_tweet 93.24.237.12jim 21 OR login 198.184.237.49marie 27 OR login 58.133.120.100marie 27 OR view_user 122.158.130.90mike 69 VA new_tweet 87.124.79.252mike 69 VA logout 55.237.104.36

  3. 优化重分区连接 传统重分区方法的实现空间效率低下。它需要将连接的所有的输出值都读取到内存中，然后进行多路连接。事实上，如果仅仅将小数据集读取到内存中，然后用小数据集来遍历大数据集，进行连接，这样将更加高效。下图是优化后的重分区连接的流程图。

  4. Map输出的组合键和组合值 (jim1,1logout 93.24.237.12) 一个map 根据value标记排序 (bob0,071CA) (jim0,021OR) (bob1,1new_tweet58.133.120.100) (jim1,1logout 93.24.237.12) (jim1,1new_tweet 93.24.237.12) 分组 另一个map (bob0,071CA) (bob1,1new_tweet58.133.120.100) (jim0,021OR) (jim1,1logout 93.24.237.12) (jim1,1new_tweet 93.24.237.12) （key，value）=（name +smaller，smaller+age+state/smaller+action+IP） jim 21 OR (jim0,021OR) jim new_tweet 93.24.237.12 (jim1,1new_tweet 93.24.237.12) bob 71 CA (bob0,071CA) bob new_tweet 58.133.120.100 (bob1,1new_tweet 58.133.120.100) 另一块： jim logout 93.24.237.12 (jim1,1logout 93.24.237.12)

  5. 影响数据整理和数据流三元素 在map输出收集阶段，由分区器选择哪个reduce应该接收map的输出。map输出的各个分区的数据，由RawComparator进行排序。Reduce端也用RawComparator进行排序。然后，由RawComparator对排序好的数据进行分组。

  6. Thank you!

More Related