PySpark Tutorial | PySpark Tutorial For Beginners | Apache Spark With Python Tutorial | Simplilearn

2. What’s in it for you? What is PySpark? PySpark Features PySpark with Python and Scala PySpark Contents PySparkSubpackages Companies using PySpark Demo using PySpark

3. What is PySpark? PySpark is the PythonAPI to support Apache Spark

4. Click here to watch the video

5. What is PySpark? PySpark is the PythonAPI to support Apache Spark + =

6. PySpark Features

7. PySpark Features Caching and disk persistence Real-time Analysis Fast processing Polyglot

8. Spark with Python and Scala Criteria Spark is written in Scala, so it integrates well and is faster than Python Python is slower than Scala when used with Spark Performance

9. Spark with Python and Scala Criteria Spark is written in Scala, so it integrates well and is faster than Python Python is slower than Scala when used with Spark Performance Python has simple syntax and being a high-level language, it’s easy to learn Scala has a complex syntax, hence is not easy to learn Learning curve

10. Spark with Python and Scala Criteria Spark is written in Scala, so it integrates well and is faster than Python Python is slower than Scala when used with Spark Performance Python has simple syntax and being a high-level language, it’s easy to learn Scala has a complex syntax, hence is not easy to learn Learning curve Code Readability Scala is a sophisticated language. Developers need to pay a lot of attention towards the readability of the code Readability, maintenance, and familiarity of code is better in Python API

11. Spark with Python and Scala Criteria Spark is written in Scala, so it integrates well and is faster than Python Python is slower than Scala when used with Spark Performance Python has simple syntax and being a high-level language, it’s easy to learn Scala has a complex syntax, hence is not easy to learn Learning curve Code readability Scala is a sophisticated language. Developers need to pay a lot of attention towards the readability of the code Readability, maintenance, and familiarity of code is better in Python API Python provides a rich set of libraries for data visualization and model building Scala lacks in providing data science libraries and tools for data visualization Data Science libraries

12. PySpark Contents

13. PySpark Contents SparkConf Broadcast & Accumulator SparkContext SparkFiles DataFrames StorageLevel RDD

14. PySpark – SparkConf

15. PySpark – SparkConf SparkConf provides configurations to run a Spark application

16. PySpark – SparkConf SparkConf provides configurations to run a Spark application The following code block has the details of a SparkConf class for PySpark class pyspark.SparkConf( loadDefaults = True, _jvm = None, _jconf = None )

17. PySpark – SparkConf SparkConf provides configurations to run a Spark application Following are some of the most commonly used attributes of SparkConf The following code block has the details of a SparkConf class for PySpark set(key, value) – To set a configuration property setMaster(value) – To set the master URL setAppName(value) – To set an application name Get(key, defaultValue=None) – To get a configuration value of a key class pyspark.SparkConf( loadDefaults = True, _jvm = None, _jconf = None )

18. PySpark - SparkContext

19. PySpark – SparkContext SparkContext is the main entry point in any Spark Program

20. PySpark – SparkContext Data Flow Local SparkContext Socket Py4J Python SparkContext Local FS JVM SparkContext is the main entry point in any Spark Program Cluster Spark Worker Spark Worker Pipe Python Python Python Python

21. PySpark – SparkContext Below code has the details of a PySpark class as well as the parameters which SparkContext can take class pyspark.SparkContext ( master = None, appName = None, sparkHome = None, pyFiles = None, environment = None, batchSize = 0, serializer = PickleSerializer(), conf = None, gateway = None, jsc = None, profiler_cls = <class ‘pyspark.profiler.BasicProfiler’> )

22. PySpark – SparkFiles

23. PySpark – SparkFiles SparkFiles allows you to upload your files using sc.addFileand get the path on a worker using SparkFiles.get

24. PySpark – SparkFiles SparkFiles allows you to upload your files using sc.addFileand get the path on a worker using SparkFiles.get SparkFiles contain the following classmethods: get(filename) getrootdirectory()

25. PySpark – SparkFiles SparkFiles allows you to upload your files using sc.addFileand get the path on a worker using SparkFiles.get getrootdirectory() specifies the path to the root directory, which contains the file that is added through the SparkContext.addFile() SparkFiles contain the following classmethods: from pyspark import SparkContext from pyspark import SparkFiles finddistance = “/home/Hadoop/examples/finddistance.R” finddistancename = “finddistance.R” sc = SparkContext(“local”, “SparkFile App”) sc.addFile(finddistance) print “Absolute path -> %s” % SparkFiles.get(finddistancename) get(filename) getrootdirectory()

26. PySpark – RDD

27. PySpark – RDD A Resilient Distributed Dataset(RDD) is the basic abstraction in Spark. It presents an immutable, partitioned collection of elements that can be operated on in parallel

28. PySpark – RDD A Resilient Distributed Dataset(RDD) is the basic abstraction in Spark. It presents an immutable, partitioned collection of elements that can be operated on in parallel RDD Transformation Action These are operations (such as reduce, first, count) that return a value after running a computation on an RDD These are operations (such as map, filter, join, union) that are performed on an RDD that yields a new RDD containing the result

29. PySpark – RDD PySpark program to return the number of elements in the RDD class pyspark.RDD ( jrdd, ctx, jrdd_deserializer = AutoBatchedSerializer(PickleSerializer()) ) from pyspark import SparkContext sc = SparkContext("local", "count app") words = sc.parallelize ( ["scala", "java", "hadoop", "spark", "akka", "spark vs hadoop", "pyspark", "pyspark and spark"] ) counts = words.count() print "Number of elements in RDD -> %i" % (counts) Creating PySpark RDD:

30. PySpark – StorageLevel

31. PySpark – StorageLevel StorageLevel decides whether RDD should be stored in the memory or should it be stored over the disk, or both RDD Disk Memory

32. PySpark – StorageLevel StorageLevel decides whether RDD should be stored in the memory or should it be stored over the disk, or both class pyspark.StorageLevel(useDisk, useMemory, useoffHeap, deserialized, replication=1) RDD from pyspark import SparkContext import pyspark sc = SparkContext( “local”, “storagelevel app” ) rdd1 = sc.parallelize([1, 2]) rdd1.persist(pyspark.StorageLevel.MEMORY_AND_DISK_2) rdd1.getStorageLevel() print(rdd1.getStorageLevel()) Disk Memory Output: Disk Memory Serialized 2x Replicated

33. PySpark – DataFrames

34. PySpark – DataFrames DataFrames in PySpark is a distributed collection of rows with named columns

35. PySpark – DataFrames DataFrames in PySpark is a distributed collection of rows with named columns Characteristics with RDD: • Immutable in nature • Lazy Evaluation • Distribution

36. PySpark – DataFrames DataFrames in PySpark is a distributed collection of rows with named columns Characteristics with RDD: Ways to create a DataFrame in Spark • It can be created using different data formats • Loading data from existing RDD • Programmatically specifying schema • Immutable in nature • Lazy Evaluation • Distribution

37. PySpark – Broadcast and Accumulator

38. PySpark – Broadcast and Accumulator A Broadcast variableallow the programmers to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks

39. PySpark – Broadcast and Accumulator A Broadcast variableallow the programmers to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks A broadcast variable is created with SparkContext.broadcast() >>> from pyspark.context import SparkContext >>> sc = SparkContext(‘local’, ‘test’) >>> b = sc.broadcast([1, 2, 3, 4, 5]) >>> b.value [1, 2, 3, 4, 5]

40. PySpark – Broadcast and Accumulator Accumulatorsare variables that are only added through an associative and commutative operation

41. PySpark – Broadcast and Accumulator Accumulatorsare variables that are only added through an associative and commutative operation class pyspark.Accumulator(aid, value, accum_param) from pyspark import SparkContext sc = SparkContext(“local”, “Accumulator app”) num = sc.accumulator(10) def f(x): global num num + = x rdd = sc.parallelize([20, 30, 40, 50]) rdd.foreach(f) final = num.value print(“Accumulated value is -> %i” % (final)) Output: Accumulated value is -> 150

42. Subpackages in PySpark

43. Subpackages in PySpark SQL Streaming ML Mllib pyspark.mllib package pyspark.streaming module pyspark.sql module pyspark.ml package

44. Companies using PySpark

45. Companies using PySpark

46. Demo using PySpark

47. Demo on Walmart Stocks data