VoD BitTorrent Framework

1. VoD BitTorrent Framework

2. Background • BT is a very popular peer to peer protocol with many implementations: • http://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clients • The first one was created in Python by Bram Cohen.

3. Real world motivation • BitTorrent is common and Free • Good chance there is a swarm for your movie • VoD servers are expensive and complicated • Its better to use P2P for VoD

4. Some definitions • Delay (D) = How much time I am allowed to hold before starting to play M (Movie). • Prefetch Time (P) = How much time before I watch byte B from M, it should arrive. • Playback Rate (R) = How many bytes/second I watch • B(t) – the index of the bit that must arrive by t. • B(t) = R *(P+t-D)

5. Rules and Examples • B(t) = R * (P+t-D) • If R=5 KB/sec and D=6 seconds and P=2 seconds, than: • At t=6 seconds, I must have 10KB continuous from the start of the movie. • At t=18 seconds I must have 70KB continuous from the start of the movie.

6. What does one VOD client do • Your client tries to get from the Bittorent clients pieces of data needed to play the video. • Each piece needs to arrive before its deadline. • If the piece does not arrive by the deadline you can download it “instantly” from the “server”. • Each VOD client counts the data from server (DFS).

7. What is my goal • Each byte taken from the server costs money, therefore you need to minimize DFS. • Group - all the VOD clients that run concurrently. • Goal - Minimize the DFS for the group.

8. How do I create my client • Download GROUP_VOD_CODE.tar.gz from “BitTornado - Start with this Code” link in the course web page. • It is also in /users/courses/hanochworkshop/group_vod • GROUP_VOD_CODE.tar.gz is a regular BT with StreamWatcher.py to monitor DFS. Any changed section is marked  ###### GROUP VOD ###### • Modify it to have lower group DFS • Do not change StreamWatched.py.

9. Standard BT parameters • --saveas F • --max_upload_rate • F.torrent

10. My VOD client new parameters • delay – Wait before prefetch, in seconds. • prefetchT – How much time in advance the byte must arrive. In percent of movie length. • rate – VOD rate in KB/sec • out_dir – Where to place the .csv files. • order – How many run when client spawned. • gap – How much to wait between spawns – unit is pewfetchT. • group_size – How many concurrent clients. • alg – Name of the algorithm. Default is ORIG. • Verbose – Print and kill the whole swarm when complete.

11. How do I test my idea • Run the environment: ./run_multiple.sh ~/GROUP_VOD_CODE/btdownloadheadless.py <GAP> ~/out • ~/GROUP_VOD_CODE/btdownloadheadless.py: Your client. • <GAP>: An integer. How many prefetchT between spawns of your clients. • ~/out: .csv files

12. Tracker BT BT BT BT BT BT BT BT BT BT VOD BT Execution example run_multiple.sh run-all.sh VOD BT VOD BT VOD BT

13. How does it work? • Run tracker • Run the “world” • Run your clients

14. How does it work? • Both tracker and the “world” run on the same computer. • Running a test: • Reserve time on pc-hanoch-w3/w4 in: • http://bt-p2p-vod.wikispaces.com/ • Or from course www - "Scheduling computers for tests.“ • cd /users/courses/hanochworkshop/group_vod/vod_tests • ./run_multiple.sh ~/btdownloadheadless.py <GAP> ~/out

15. Output to screen ================================================================================== total time: 1 min 07 sec share rating: 0.904 (4816.0 KB up / 5328.0 KB down) saving: picture-100.bmp (27.5 MB) percent done: 18.9 time left: 4 min 52 sec download to: /home/pc-hanoch-w4/home/hillelav/name/picture.pc-hanoch-w4-4.bmp download rate: 93.4 kB/s upload rate: 80.6 kB/s seed status: 2 seen now, plus 0.340 distributed copies peer status: 28 seen now, 10.7% done at 1220.3 kB/s --------------------------------StreamWatcher------------------------------------- Csv stats: ~/out/1/statistics-order-4-gap-0.csv DFS is: 8372224 bytes DFS/Total is: 29 % FullPieces: 63 / 220 DirtyPieces: 1 / 220 FaildHashChecks: 0 Prefetching 5 % Playing point: 56 / 220 ( 25 %) Only last VOD client prints

16. Output to .csv • In ~/out you will find 4 directories: /1, /2, /3 and /4. • In each directory, you have 4 files • statistics-order-<1..4>-gap-<GAP>.csv • Order is how many VOD clients ran before. • <GAP> is the parameter <GAP>.

17. .csv file alg,dfs,p2p ORIG,5,0 ORIG,10,2 ORIG,15,6 ORIG,19,11 ORIG,23,16 ORIG,28,20 ORIG,31,25 ORIG,34,31 ORIG,37,36 ORIG,39,41 . . .

18. Required plot 1 - total DFS and total P2P – GAP = 2 File percentage File percentage

19. GAP = 0

20. Required plot 2 - local DFS GAP = 2

21. GAP = 0

22. The main script - run_multiple.sh • Location: • /users/courses/hanochworkshop/group_vod/vod_tests • run_multiple.sh: • Runs run_all.sh four times. • Parameters: • Your client – PATH/btdownloadheadless.py • Gap – wait Gap*PrefetchT between VOD spawns. • Csv location – where to store the .csv files.

23. run_all.sh defaults • Start • VODClient: • Max_upload rate = 90 • prefetchT = 5% • Rate = 90 • Alg = ORIG • Verbose = TRUE # For last client in group • Stop • Kill all python threads • Kill • Kill all python threads and scripts.

24. To run client separately: • PATH/btdownloadheadless.py --saveas F --delay D --prefetchT P --rate R --out_dir O Can add the parameters --order X in the IDE --gap G --group_size S --alg ‘ALG' --verbose F.Torrent

25. Python • Very common script language • IDE: • PyDev • WINGWARE Python IDE – not free, but very good. • References: • python.org :http://docs.python.org/reference/ • http://wiki.python.org/moin/BeginnersGuide/NonProgrammers • http://wiki.python.org/moin/BeginnersGuide/Programmers • The book "Think like a Computer Scientist in Python" has a very good reputation:

26. Getting started • Run the environment • ssh pc-hanoch-w3 or pc-hanoch-w4 • cd /users/courses/hanochworkshop/group_vod/vod_tests • run_multiple.sh MyClient GAP output-directory • Note tracker prints a message for each GET. • You can connect to the tracker, from the CS network, with a browser at: • http://pc-hanoch-wk:6970/ • Do ls -l /home/pc-hanoch-wk/home/hillelav/$USER to see downloaded files

27. Directory Structure • Root – • high level application • Root/BitTornado – • services • Root/BitTornado/BT1 – • Actual work

28. Client Operation • btdownloadheadless.py – First to run. • Performs initializations. • get_response – parse the .torrent • Infohash – the ID of the .torrent • startEngine – Start connection with peers • startRerequester - Start connection with tracker • Run rawserver.listen_forever from RawServer.py. • listen_forever is: • Polling port • executing func() from a task queue.

29. Receiving messages • From listen_forever loop it goes to: • BT1/Connecter.py/got_message • CHOKE • UNCHOKE • INTERESTED • NOT_INTERESTED • BITFIELD • REQUEST • CANCEL • PIECE • HAVE

30. Influencing decisions • Can be done in one of the callbacks which spring from Connecter.py / got_message: • Downloader.py • HAVE->got_have->send_interested • Tune for VoD?

31. BT tracker • Send GET request in Rerequester.py / announce • Receive GET data in thread in Rerequester.py / _rerequest

32. Policy • Choker.py/_round_robin – decide who to choke / optimistically un choke / send_have to. • Downloader.py / _request_more. • PiecePicker.py / next - rarest piece, etc. Tune for VoD? • got_unchoke->_request_more->next…

33. More about the code • Start your project from “BitTornado - Start with this Code” link. • Click “BitTornado - Legacy Code Guide” for basic explanations about the code.