AWS Data Pipeline Tutorial | AWS Tutorial For Beginners | AWS Certification Training | Edureka

2. Agenda 02 04 01 03 What is AWS Data Pipeline components Demo on AWS Data Pipeline Need for Data Pipeline AWS Data Pipeline? AWS Architect Certification Training www.edureka.co/cloudcomputing

3. Need of Data Pipeline Copyright © 2018, edureka and/or its affiliates. All rights reserved.

4. A Hypothetical Example Goal 2 Manage application efficiently Goal 3 Improve business faster but at cheaper rate Goal 1 Improve business by targeting content AWS Architect Certification Training www.edureka.co/cloudcomputing

5. Problem Statement Huge amount of data in different formats – so processing, storing & migrating data becomes complex Real-time data for registered users DynamoDB Webserver logs for potential users Amazon S3 Amazon RDS Demographic data & login credentials Sensor data & 3rd party datasets Amazon S3 AWS Architect Certification Training www.edureka.co/cloudcomputing

6. Solution Feasible Solution Optimal Solution Use a data pipeline which handles processing, visualisations & migration Analyse the data & convert from unstructured to structured format AWS Architect Certification Training www.edureka.co/cloudcomputing

7. AWS Data Pipeline Copyright © 2018, edureka and/or its affiliates. All rights reserved.

8. AWS Data Pipeline AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on- premises data sources, at specified intervals. AWS Data Pipeline Weekly task Daily task Task: Launch data analysis Task: Copy log files S3 Bucket Amazon EMR EC2 Instance AWS Architect Certification Training www.edureka.co/cloudcomputing

9. AWS Data Pipeline DATA DATA Data Stores (I/P) Compute Resources Data Stores (O/P) AWS Architect Certification Training www.edureka.co/cloudcomputing

10. Example: Launch Data Analysis Collect data from different data sources, perform EMR analysis & generate weekly reports Event data from DynamoDB Daily EMR results Daily EMR analytics Bulk data from S3 Weekly report in Redshift Weekly summary report AWS Architect Certification Training www.edureka.co/cloudcomputing

11. Benefits of AWS Data Pipeline Benefits of AWS Data Pipeline Full control over the computational resources Provides a drag & drop console Built on distributed, reliable infrastructure Inexpensive to use Distribute work to one machine or many Supports scheduling & error handling AWS Architect Certification Training www.edureka.co/cloudcomputing

12. AWS Data Pipeline Components Copyright © 2018, edureka and/or its affiliates. All rights reserved.

13. Components of AWS Data Pipeline Specifies business logic of data management Pipeline definition Activities Schedules Data Nodes Pipeline Task Runner Resources Preconditions Actions AWS Architect Certification Training www.edureka.co/cloudcomputing

14. Components of AWS Data Pipeline Schedules & runs the tasks to perform defined activities Pipeline definition Pipeline Pipeline Components Instances Attempts Task Runner AWS Architect Certification Training www.edureka.co/cloudcomputing

15. Components of AWS Data Pipeline Polls AWS Data Pipeline for tasks & then performs those tasks Pipeline definition No Retries remaining Did task succeed? Pipeline schedules tasks Task ends Pipeline Yes Yes No Task Runner Task runner reports when task is done Task runner polls tasks AWS Architect Certification Training www.edureka.co/cloudcomputing

16. Demo – Import & export DynamoDB data Copyright © 2018, edureka and/or its affiliates. All rights reserved.

17. AWS Architect Certification Training www.edureka.co/cloudcomputing