PySpark Programming | PySpark Concepts with Hands-On | PySpark Training | Edureka

1. PySpark Tutorial Copyright © 2018, edureka and/or its affiliates. All rights reserved.

2. Objectives of Today’s Training PySpark RDDs DataFrame Programming PySpark SQL PySpark Streaming Machine Learning (MLlib) Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

3. PySpark Copyright © 2018, edureka and/or its affiliates. All rights reserved.

4. PySpark Visualization is Possible Python A PI for Spark W ide Range of L ibraries U ses Py4j to launch JVM Sim ple A PI E asy to L earn & U se Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

5. RDDs Copyright © 2018, edureka and/or its affiliates. All rights reserved.

6. Resilient Distributed Dataframe (RDD) RDD is the abstracted data over the distributed collection Created using various Spark Context Functions Follows lazy initialization principle RDDs are immutable and cacheable in nature Transformations Supports two different types of operations Actions Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

7. RDD – Transformations & Actions Transformations Actions Map(func) take(N) flatMap(func) count() filter(func) collect() groupByKey() reduce() reduceByKey(func) takeOrdered(N) mapValues(func) top(N) Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

8. DataFrame Copyright © 2018, edureka and/or its affiliates. All rights reserved.

9. DataFrame Immutable but distributed collection of structured & semi- structured data 1 2 Organized into named columns similar to a RDMS table 3 Helps in increase in performance of PySpark queries 4 Supports a wide range of data formats and sources 5 API support for various languages like Python, R, Scala, Java Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

10. PySpark SQL Copyright © 2018, edureka and/or its affiliates. All rights reserved.

11. PySpark SQL 01 03 PySparkSQL is used for processing structured and semi-structured datasets PySparkSQL provides an optimized API Through PySparkSQL, SQL and HiveQL code can be used PySparkSQL module is a higher-level abstraction over PySpark Core 02 04 Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

12. PySpark Streaming Copyright © 2018, edureka and/or its affiliates. All rights reserved.

13. PySpark Streaming PySpark Streaming is the structured stream processing framework that utilizes Spark DataFrames Library APIs Discretized Stream Fault Tolerant PySpark Streaming is the live data streaming library of PySpark It is a set of APIs that provide a wrapper over PySpark Core Discretized Stream or Dstream is a high-level abstraction which represents a continuous stream of data It can efficiently deal with various fault-tolerance aspects and is highly scalable Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

14. PySpark Streaming Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

15. PySpark Streaming Spark Streaming receives live input data streams and divides the data into batches Engine Input Stream Data Batches of Input Data Batches of Processed Data Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

16. Machine Learning Copyright © 2018, edureka and/or its affiliates. All rights reserved.

17. Machine Learning (MLlib) PySpark facilitates the development of custom ML algorithms MLlib in PySpark, is a machine-learning library It is a wrapper over PySpark Core to do data analysis using machine-learning algorithms It works on distributed systems and is scalable Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

18. Machine Learning (MLlib) MLlib provides three core machine learning functionalities 01 02 03 Data preparation Machine learning algorithms Utilities Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

19. Machine Learning (MLlib) MLlib provides three core machine learning functionalities 01 02 03 Data preparation Machine learning algorithms Utilities Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

20. Machine Learning (MLlib) MLlib provides three core machine learning functionalities 01 02 03 Data preparation Machine learning algorithms Utilities Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

21. Machine Learning (MLlib) MLlib provides three core machine learning functionalities 01 02 03 Data preparation Machine learning algorithms Utilities Python Spark Certification Training using PySpark www.edureka.co/pyspark-certification-training

22. @ Copyright © 2018, edureka and/or its affiliates. All rights reserved.