VoD BitTorrent Framework

1. VoD BitTorrent Framework

2. Background • BT is a very popular peer to peer protocol with many implementations: • http://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clients • The first one was created in Python by Bram Cohen.

3. Real world motivation • BitTorrent is common and Free • Good chance there is a swarm for your movie • VoD servers are expensive and complicated • Its better to use P2P for VoD

4. Some definitions • Delay (D) = How much time I am allowed to hold before starting to play M (Movie). • Prefetch Time (P) = How much time before I watch byte B from M, it should arrive. • Playback Rate (R) = How many bytes/second I watch • B(t) – the index of the bit that must arrive by t. • B(t) = R *(P+t-D)

5. Rules and Examples • B(t) = R * (P+t-D) • If R=5 KB/sec and D=6 seconds and P=2 seconds, than: • At t=6 seconds, I must have 10KB continuous from the start of the movie. • At t=18 seconds I must have 70KB continuous from the start of the movie.

6. What does one VOD client do • Your client tries to get from the Bittorent clients pieces of data needed to play the video. • Each piece needs to arrive before its deadline. • If the piece does not arrive by the deadline you can download it “instantly” from the “server”. • Each VOD client counts the data from server (DFS).

7. What is my goal • Each byte taken from the server costs money, therefore you need to minimize DFS. • Goal - Minimize the accumulated DFS of all the VOD clients that run concurrently.

8. How do I create my client • Download GROUP_VOD_CODE.tar.gz from “BitTornado - Start with this Code” link in the course web page. • It is also in /users/courses/hanochworkshop/group_vod • GROUP_VOD_CODE.tar.gz is a regular BT with StreamWatcher.py to monitor DFS. Any changed section is marked  ###### GROUP VOD ###### • Modify it to minimize acumulated DFS. • Do not change StreamWatched.py.

9. Standard BT parameters • --saveas F • --max_upload_rate • F.torrent

10. VOD client additional params • SET = defined inside run_all.sh, not a parameter. • delay – Wait before prefetch, in seconds. SET TO 20 • prefetchT – How much time in advance the byte must arrive. In percent of movie length. SET TO 5% • rate – VOD rate in KB/sec. SET TO 90 KB. • order – How many run when client spawned. SET BY run_all.sh. • Parameter of run_multiple.sh. • gap – How much to wait between spawns – unit is pewfetchT. • alg – Name of the algorithm. Default is ORIG. • verbose – Print and kill the whole swarm when complete. • peers_num – how many regular peers. • seeds_num – how many seeds. • Vod_num – how many VoD peers.

11. How do I test my idea • cd /users/courses/hanochworkshop/voddiversegroup/vod_tests • run_multiple.sh • your_client - example: ~/GROUP_VOD_CODE/btdownloadheadless.py • Gap – example: 0 • output_location example: ~/bt_out • iteration example: 4 • empty/full example: empty • peers_num example: 16 • seeds_num example: 2 • vod_num example: 14

12. Tracker BT BT BT BT BT BT BT BT BT BT VOD BT Execution example run_multiple.sh run-all.sh VOD BT VOD BT VOD BT

13. How does it work? • Run tracker • Run the “world” • Run your clients

14. How does it work? • Both tracker and the “world” run on the same computer. • Running a test: • Reserve time on pc-hanoch-w3/w4 in: • http://bt-p2p-vod.wikispaces.com/ • Or from course www - "Scheduling computers for tests.“

15. Output to screen ================================================================================== total time: 1 min 07 sec share rating: 0.904 (4816.0 KB up / 5328.0 KB down) saving: picture-100.bmp (27.5 MB) percent done: 18.9 time left: 4 min 52 sec download to: /home/pc-hanoch-w4/home/hillelav/name/picture.pc-hanoch-w4-4.bmp download rate: 93.4 kB/s upload rate: 80.6 kB/s seed status: 2 seen now, plus 0.340 distributed copies peer status: 28 seen now, 10.7% done at 1220.3 kB/s --------------------------------StreamWatcher------------------------------------- Csv stats: ~/out/1/statistics-order-4-gap-0.csv DFS is: 8372224 bytes DFS/Total is: 29 % FullPieces: 63 / 220 DirtyPieces: 1 / 220 FaildHashChecks: 0 Prefetching 5 % Playing point: 56 / 220 ( 25 %) Only last VOD client prints

16. Output to .csv • In ~/out you will find vod_num directories: /1, /2, …/iteration. • In each directory, you have vod_num files. • Statistics in - order-<1..vod_num>-gap-<GAP>.csv • Order is how many VOD clients ran before. • <GAP> is the parameter Gap.

17. .csv file alg,dfs,p2p ORIG,5,0 ORIG,10,2 ORIG,15,6 ORIG,19,11 ORIG,23,16 ORIG,28,20 ORIG,31,25 ORIG,34,31 ORIG,37,36 ORIG,39,41 . . .

18. total DFS and total P2P GAP = 2 File percentage File percentage

19. GAP = 0

20. Local DFS GAP = 2

21. GAP = 0

22. run_all.sh defaults • Start • Use in run_multiple.sh • Stop • Kill all python threads • Kill • Kill all python threads and scripts.

23. To run client separately: • PATH_TO_YOUR_VOD_CLIENT/btdownloadheadless.py --saveas F --delay D --prefetchT P --rate R --out_dir O Can add the parameters --order X in the IDE --gap G --group_size S --alg ‘ALG' --verbose F.Torrent

24. Python • Very common script language • IDE: • PyDev • WINGWARE Python IDE – not free, but very good. • References: • python.org :http://docs.python.org/reference/ • http://wiki.python.org/moin/BeginnersGuide/NonProgrammers • http://wiki.python.org/moin/BeginnersGuide/Programmers • The book "Think like a Computer Scientist in Python" has a very good reputation:

25. Getting started • Run the environment • ssh pc-hanoch-w3 or pc-hanoch-w4 • cd /users/courses/hanochworkshop/group_vod/vod_tests • run_multiple.sh MyClient GAP output-directory • Note tracker prints a message for each GET. • You can connect to the tracker, from the CS network, with a browser at: • http://pc-hanoch-wk:6970/ • Do ls -l /home/pc-hanoch-wk/home/hillelav/$USER to see downloaded files

26. Directory Structure • Root – • high level application • Root/BitTornado – • services • Root/BitTornado/BT1 – • Actual work

27. Client Operation • btdownloadheadless.py – First to run. • Performs initializations. • get_response – parse the .torrent • Infohash – the ID of the .torrent • startEngine – Start connection with peers • startRerequester - Start connection with tracker • Run rawserver.listen_forever from RawServer.py. • listen_forever is: • Polling port • executing func() from a task queue.

28. Receiving messages • From listen_forever loop it goes to: • BT1/Connecter.py/got_message • CHOKE • UNCHOKE • INTERESTED • NOT_INTERESTED • BITFIELD • REQUEST • CANCEL • PIECE • HAVE

29. Influencing decisions • Can be done in one of the callbacks which spring from Connecter.py / got_message: • Downloader.py • HAVE->got_have->send_interested • Tune for VoD?

30. BT tracker • Send GET request in Rerequester.py / announce • Receive GET data in thread in Rerequester.py / _rerequest

31. Policy • Choker.py/_round_robin – decide who to choke / optimistically un choke / send_have to. • Downloader.py / _request_more. • PiecePicker.py / next - rarest piece, etc. Tune for VoD? • got_unchoke->_request_more->next…

32. More about the code • Start your project from “BitTornado - Start with this Code” link. • Click “BitTornado - Legacy Code Guide” for basic explanations about the code.