Multiprocessing, Multithreading, and Async in Python: How to Do Multiple Things at the Same Time?

1. Trilha – Python Jeff Andrade Developer - CI&T

2. Multiprocessing, Multithreading e Async • Aiohttp • Serviços real-time • Long-polling • SSE • Websockets

3. Como fazer mais de uma coisa ao mesmo tempo com Python? Multiprocessing Multithreading Asyncio

4. Como funciona? Multiprocessing: Gerenciado pelo sistema operacional. A concorrência é com base na quantidade de núcleos do processador (muti-core). Multithreading: Gerenciado pelo sistema operacional. No caso do Cpython, é executado em um único núcleo do processador (não suporta multi-core). Async não depende do sistema operacional. Utiliza um processo, uma thread. O gerenciamento se dá através de um scheduler chamado de Event Loop.

5. Escalabilidade Multiprocessing: 1 → Quantidade de processadores (núcleos) Multithreading: Dezenas, centenas. Concorrência através do tempo de processamento. Async: Milhares → Milhões ?

6. Desafio de Xadrez - Miguel Grinberg Cenário 2 Cenário 1 Desafiante x Jogador 1, 2 e 3 Desafiante x Jogador 1 6 minutos Desafiante x Jogador 2 Desafiante x Jogador 3 15 minutos

7. Generators Python 3.5+ Python 3 import asyncio async def say_hello(word): print(f’hello {word}’) async def main(): await say_hello(“world”) asyncio.run(main()) import asyncio loop = asyncio.get_event_loop() @asyncio.coroutine def hello(word): print('hello {}'.format(word) @asyncio.coroutine def main(): yeld from hello(‘world’) if __name__ == ‘__main__’: asyncio.run_until_complete(main())

8. Síncrono

9. import time def hold(sec): print(f'Running for {sec} seconds') return time.sleep(sec) def main(): _start = time.time() hold(3) print(f"Execution time: { time.time() - _start }") if __name__ == '__main__': main()

10. Síncrono - Multitasking

11. import time from multiprocessing import Process def hold(sec): print(f'Running for {sec} seconds') return time.sleep(sec) def main(): _start = time.time() times = [2, 3, 4] processes = [] for seconds in times: process = Process(target=hold, args=(seconds,)) processes.append(process) for process in processes: process.start() for process in processes: process.join() print(f"Execution time: { time.time() - _start }") if __name__ == '__main__':

12. Assíncrono

13. Assíncrono import asyncio import sys import time import aiohttp import async_timeout async def hold(sec): print(f'Running for {sec} seconds') await asyncio.sleep(sec) async def main(): await hold if __name__ == '__main__': _start = time.time() loop = asyncio.get_event_loop() loop.run_until_complete(main) print(f"Execution time: { time.time() - _start }") loop.close()

14. Assíncrono - Multitasking

15. import asyncio import sys import time import aiohttp import async_timeout async def hold(sec): print(f'Running for {sec} seconds') await asyncio.sleep(sec) async def main(): times = [2, 3, 4] tasks = [] for seconds in times: task = asyncio.ensure_future(hold(seconds)) tasks.append(task) await asyncio.gather(*tasks) if __name__ == '__main__': _start = time.time() loop = asyncio.get_event_loop() future = asyncio.ensure_future(main()) loop.run_until_complete(future) print(f"Execution time: { time.time() - _start }") loop.close()

16. Serviços real-time

17. Serviços real-time Long-polling Websockets Server-sent Events

18. Como funciona?

19. FROM python:3.7 ENV PYTHONUNBUFFERED 1 ENV C_FORCE_ROOT true # Executa como root WORKDIR /src ADD ./src /src RUN pip install --upgrade pip RUN pip install -r requirements.txt CMD gunicorn server:factory --bind 0.0.0.0:8000 --worker-class aiohttp.GunicornWebWorker

20. import asyncio import json from aiohttp.web import Application, Response from aiohttp_sse import sse_response import aiohttp_cors import sockjs score = {'cruzeiro': 0, 'atletico': 0} ws = [] async def factory(): loop = asyncio.get_event_loop() app = Application(loop=loop) cors = aiohttp_cors.setup(app, defaults={ "*": aiohttp_cors.ResourceOptions( allow_credentials=True, expose_headers="*", allow_headers="*", ) }) cors.add(app.router.add_route('POST', '/vote_cruzeiro', vote_cruzeiro)) cors.add(app.router.add_route('POST', '/vote_atletico', vote_atletico)) cors.add(app.router.add_route('GET', '/sse/', sse_poll)) sockjs.add_endpoint(app, ws_poll, name='ws', prefix='/ws/') return app

21. async def vote_cruzeiro(request): score['cruzeiro'] += 1 for ws_session in ws: data = json.dumps(score) ws_session.manager.broadcast(data) return Response() async def vote_atletico(request): score['atletico'] += 1 for ws_session in ws: data = json.dumps(score) ws_session.manager.broadcast(data) return Response()

22. async def sse_poll(request): async with sse_response(request) as resp: while True: data = json.dumps(score) await resp.send(data) await asyncio.sleep(0)

23. async def ws_poll(msg, session): if msg.type == sockjs.MSG_OPEN: ws.append(session) for ws_session in ws: data = json.dumps(score) ws_session.manager.broadcast(data) print("Someone joined") elif msg.type == sockjs.MSG_MESSAGE: session.manager.broadcast(msg.data) elif msg.type == sockjs.MSG_CLOSED: ws.remove(session) print("Someone left.")