JobRunner

1. JobRunner Running Distributed tasks using Python and AMQP Gabriel Samfira, Cloud Engineer

2. A bit of background Cloudbase has seen significant growth since starting work on OpenStack Testing is essential The need for automating all this madness has become painfully obvious

3. JobRunner Distribute workload Allow users to create own tasks Light weight Decently extensible

4. Components JobRunner REST API (sort of) JobRunner Controller JobRunner Worker JobRunner Notifications

5. Design Evented worker or single threaded All interactions between components are asynchronous Tasks are created for resources Dispatcher determines the module that should be loaded Actions define the method from the module (well..actually class) that is to be run

6. Resources At a minimum, all resources must define the following methods: create delete is_idle lock Save And the following attributes: uuid (this would be the task ID) state

7. Tasks A few of the more relevant attributes of a task: (yes, there are more, but not many more) task_id queue action resource state body <– The contents of this little bugger is sent to the workers result traceback error snapshot

8. Task body { "resource": "jobs", "action": "create", "body": { "job_args": "[\"http://example.com/img.img\", \"xxxxxsha1xxxxx\"], "job_name": "test_job", "job_id": "5ed1361fd3bb4101a7457011a6e5a4d7" } }

9. JobRunner dispatcher defdispatch(message): # Must return a tuple in the form of (result,state,error,strace) msg_body = json.loads(message.body) log.debug("Got message. Processing: %s"% msg_body) resource = msg_body['resource'] mod = importlib.import_module( 'jrunner.jobworker.callbacks.%s'%str(resource)) ret = mod.process(msg_body) # Send reply here rsp = build_response(*ret) send_reply(rsp, message) returnTrue

10. Taskset layout

12. https://github.com/gabriel-samfira/jrunner

13. Questions?