大规模数据处理 / 云计算 Lecture 2 – "Hello World" in Hadoop

1. 大规模数据处理/云计算Lecture 2 – "Hello World" in Hadoop 彭波 北京大学信息科学技术学院 7/3/2014 http://net.pku.edu.cn/~course/cs402/ Jimmy Lin University of Maryland SEWMGroup This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United StatesSee http://creativecommons.org/licenses/by-nc-sa/3.0/us/ for details

2. CodeLab1 • 遇到的困难 • 不熟悉java！ • 开发和运行环境搭建？（eclipse, hadoop) • guide里面的代码编译报错？ • 运行时报错？ • 。。。。。。。。。

3. 貌似pdf里给的代码不能用，点那个“source code here”出来的代码是能用的……呃……不过我跑出来的结果和pdf里的不一样…… • The method setInputPath(Path) is undefined for the type JobConf WordCount/src WordCount.java line 21 1404272734726 310不知道什么原因。。 • 编译通不过 求助 • FileInputPath cannot be resolvedFileOutputPath cannot be resolved这是什么情况。 • Exception in thread "main" java.io.IOException: Cannot run program "chmod": CreateProcess error=2, ?????????我运行的时候报的这个错误

4. Java Programming for C/C++ Developers

5. Historical background • The C programming language • early 1970s • UNIX • The C++ programming language • early 1980s • object-oriented • a wide variety of application programming • The Java programming language • early 1990s • originally for consumer electronic devices • enterprise application development

6. Java SDK • Software Development Kit • a group of command-line tools and packages that you will need to write and run Java programs • base classes (Library)

7. Working with the SDK • Factorial • input: a value as a command-line argument • output: factorial of that number OR exception • Java Specification • every Java source code file must have the exact same name as the class that is defined inside of it

8. Execution Environment

9. Primitive data types • Char • 16 bits • Unicode character set • escape sequences

10. Primitive data types • integer types • signed • exact size

11. Primitive data types • The floating-point types • IEEE 754 floating-point values

12. Primitive data types • The boolean types • true, false

13. Operators • + is overloaded • If you use the + operator with a String and another operand that is not a String, the other operand is converted into a String

14. C/C++ functions versus Java methods • In Java terminology, functions are called methods. • Methods can only be declared as members of a class; you can't define a method outside of a Java class

15. Arrays • objects, so they are declared using the new operator • scores.length • the bracket characters ([ ]) that are used to indicate arrays are bound to the array type, not the array name • java.lang.ArrayIndexOutOfBounds exception

16. Strings • objects of the String class • String objects are immutable • same string literals • String class has a rich interface

17. Strings

18. The main() method • a strict naming convention • first element in the array is the first argument, not the name of the program.

19. Other differences • Pointers: • Java references are pointers to Java objects • cannot be incremented or decremented • no address of operators • Global variables • no way to declare global variables (or methods) • no struct, union, typedef, enum • Freely placed methods • Garbage collection • no malloc() and free()

20. Defining a Java class

23. Defining a Java class • Each member must have its own public or private modifier • You don't use semicolons (;) after the closing brackets in class and method definitions. • The main() method is a member of the class • You call the constructor using the new keyword

24. access modifiers

25. access modifiers • public • private • protected • package access

26. Inheritance • extends • super()

27. Overloading and overriding

28. The Object class • All Java classes are ultimately subclasses of class Object • a centrally rooted class hierarchy • usage • toString() • define data structures that take objects of class Object , it can hold any Java object .vs. C++ template

29. Interfaces • All interfaces are implicitly abstract • All members of an interface are implicitly public • All fields defined in an interface are implicitly static and final • A Java class can extend only one class, but it can implement any number of interfaces • Best practice for polymorphism

31. more on objects • Inner classes and inner interfaces • Anonymouse classes and objects

33. Using Library(Java API) • Java API, classes are grouped into packages • you already been using classes from a default package: java.lang when call System.out.println() • import java.util.ArrayList; or java.util.ArrayList<xx> list = ....

34. Data Structures • java.util.* • java generics

35. Deploying your application • A Java program is a bunch of classes. • A JAR file is Java Archive • create a manifest.txt state which class has main() method • Main-Class: MyApp • use jar tool to package all classes files and manifest.txt • $jar -cvmf manifest.txt app.jar *.class • $java -jar app.jar

36. Package • put your classes in packages • java.util, java.net, java.text .... • preface your package with your reverse domain name • setup a matching directory structure

37. References • 《Java programming for C C++ developers》 • 《Head First Java》

39. "Hello World" in Hadoop

40. What is MapReduce? Programming model for expressing distributed computations at a massive scale Execution framework for organizing and performing such computations Open-source implementation called Hadoop 40

41. Brief History of Hadoop • Hadoop was created by Doug Cutting, the creator of Apache Lucene/Nutch, • 2003, Google published GFS • 2004, Google published MapReduce • 2005, Nutch ported to Mapreduce/HDFS • 2006, Cutting join Yahoo! • 2008.1, Hadoop became top-level project at Apache • 2008.2, Hadoop run on 10000-core cluster

42. Hadoop Release

43. New MapReduce API • favors abstract classes over interfaces • new API in org.apache.hadoop.mapreduce, old in org.apache.hadoop.mapred • new Context class • JobConf, OutputCollector,Reporter • new Job class • JobClient • reduce() method passes values • new: java.lang.Iterable, for (VALUEIN value : values) { ... } • old: java.lang.Iterator, hasNext(), next()

44. Hadoop Streaming & Pipes • Streaming • support any programming language, even shell scripts • uses standard input and output to communicate with the map and reduce code • Pipes • C++ interface to Hadoop MapReduce • uses sockets as the communication channel

45. Hadoop Command • docs in distribution • api • tutorial • hadoop • -conf xxx

46. Changping Cluster • 28 Nodes, 12 Cores/48GB RAM/10T DISK • Namenode/JobTracker server - changping11 • ip : 222.29.134.11 • hdfs port : 9000 • mapreduce port: 9001

47. How to use ChangpingCluster • 1. 添加一个域名解析 • windows: 编辑 C:\WINDOWS\system32\drivers\etc\hosts 文件, • linux : /etc/hosts 添加一行如下: 222.29.134.11 changping11 • 否则运行 job 会报告名字解析错误

48. How to use ChangpingCluster • 2. 身份设置 • 1). 输出文件统一到 "/cs402/YourName"目录下 • 代码中是：FileOutputFormat.setOutputPath(conf, new Path("/cs402/YourName")); • 2). Mapred Location里设置好hadoop.job.ugi = YourName, cs402 • 用户名和上面文件路径中的名字一致， • 组名必须是 cs402 • 或者在driver程序里直接设置好。 • Configuration conf = new Configuration(); • conf.set("hadoop.job.ugi", "YourName,cs402");

49. References • Tom White, Hadoop: The Definitive Guide, O'Reilly, 3rd, 2012.5.

50. Q&A