1 / 7

Cloud-based Architecture for Batch Processing Systems

Explore the power of Cloud Architectures with AWS for processing pipelines, nightly builds, and websites. Utilize Amazon EC2, S3, SQS, and SimpleDB for efficient and scalable data management. Architectural abstractions detailed at three levels for submission, processing, and monitoring of jobs.

gdecker
Download Presentation

Cloud-based Architecture for Batch Processing Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Analysis and Design AWS Submitter

  2. ETICS SubmissionManager and AWS • Power of Cloud Architectures • From “GrepTheWeb-AWS white paper” • Processing Pipelines • Batch processing systems • Nightly builds, automated unit/deployment testing • Websites Amazon AWS • Amazon EC2 for running a large distributed processing cluster on-demand • Amazon S3 for retrieving/storing persistent datasets • Amazon SQS for buffering requests - loosely coupled • Amazon SimpleDB for storing data - status, log

  3. Architectural Abstraction : Level 1 • AWS as a Grid Input System • sendJob : send ETICS jobs to Amazon Cloud • getStatus : receive information about the status of ETICS job. Output System • GetOutput : result of processing ETICS jobs.

  4. sendJob getOutput getSatus Architectural Abstraction : Level 2

  5. Architectural Abstraction : Level 2 • The ETICS Submitter • send a message in to the SQS queue for each request The controller (EC2 instance): • takes a message from the SQS queue • use SimpleDB to store Job information • launches EC2 instances (worker nodes) to run the job • monitor the job status • Store the output on S3

  6. Architectural Abstraction : Level 3

  7. On application start SQS queue creation and start all the controller Launch Controller pick up message from launch queue, populated by ETICS send the job request to EC2 cluster update status in SimpleDB (ready) delete the message in the launch queue insert a message in monitor queue Monitor Controller pick up message from monitor queue ping EC2 cluster to check the job status take/update the status in SimpleDB (running) insert a message in delete queue Shutdown Controller shutdown EC2 instances pick up message from shutdown queue update the status in SimpleDB delete a message after processing Architectural Abstraction : Level 3

More Related