70 likes | 82 Views
Analysis and Design. AWS Submitter. ETICS SubmissionManager and AWS. Power of Cloud Architectures From “GrepTheWeb-AWS white paper” Processing Pipelines Batch processing systems Nightly builds, automated unit/deployment testing Websites Amazon AWS
E N D
Analysis and Design AWS Submitter
ETICS SubmissionManager and AWS • Power of Cloud Architectures • From “GrepTheWeb-AWS white paper” • Processing Pipelines • Batch processing systems • Nightly builds, automated unit/deployment testing • Websites Amazon AWS • Amazon EC2 for running a large distributed processing cluster on-demand • Amazon S3 for retrieving/storing persistent datasets • Amazon SQS for buffering requests - loosely coupled • Amazon SimpleDB for storing data - status, log
Architectural Abstraction : Level 1 • AWS as a Grid Input System • sendJob : send ETICS jobs to Amazon Cloud • getStatus : receive information about the status of ETICS job. Output System • GetOutput : result of processing ETICS jobs.
sendJob getOutput getSatus Architectural Abstraction : Level 2
Architectural Abstraction : Level 2 • The ETICS Submitter • send a message in to the SQS queue for each request The controller (EC2 instance): • takes a message from the SQS queue • use SimpleDB to store Job information • launches EC2 instances (worker nodes) to run the job • monitor the job status • Store the output on S3
On application start SQS queue creation and start all the controller Launch Controller pick up message from launch queue, populated by ETICS send the job request to EC2 cluster update status in SimpleDB (ready) delete the message in the launch queue insert a message in monitor queue Monitor Controller pick up message from monitor queue ping EC2 cluster to check the job status take/update the status in SimpleDB (running) insert a message in delete queue Shutdown Controller shutdown EC2 instances pick up message from shutdown queue update the status in SimpleDB delete a message after processing Architectural Abstraction : Level 3