1 / 53

Getting data home

Learn various methods to transfer data home securely, including FTP, tapes, and troubleshooting tips for compatibility issues. Improve your workflow and avoid distractions.

ceciliag
Download Presentation

Getting data home

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Getting data home ?

  2. Getting data home ? FTP

  3. Getting data home ? FTP tapes

  4. Getting data home ? FTP: not secure tapes: incompatible

  5. Getting data home ? FTP: not secure tapes: incompatible distracting

  6. Getting data home ? http://bl831.als.lbl.gov/~jamesh/beamline/moving_data.html

  7. Getting data home ? http://bl831.als.lbl.gov/~jamesh/beamline/moving_data.html http://bcsb-web.als.lbl.gov/beamline/Sector8/DataBackup.htm

  8. Getting data home ? http://bl831.als.lbl.gov/~jamesh/beamline/moving_data.html http://bcsb-web.als.lbl.gov/beamline/Sector8/DataBackup.htm http://ucxray.berkeley.edu/~jamesh/elves/manual/tricks.html

  9. Can you login? ssh user@home

  10. Universal directory-transfer tar cf - . | ssh user@home “cd /dir ; tar xvf -”

  11. Universal directory-transfer tar cf - . | ssh user@home “cd /dir ; tar xvf -”

  12. Universal directory-transfer tar cf - . | ssh user@home “cd /dir ; tar xvf -” Password: Ctrl-z bg

  13. Making it faster tar cf - . | compress | ssh user@home “cd /dir ; uncompress | tar xvf -”

  14. The easy way http://bl831.als.lbl.gov/~jamesh/elves/scripts/sendhome sendhome . user@home:/dir

  15. Real Time http://bl831.als.lbl.gov/~jamesh/elves/scripts/sendhome sendhome first_001.img user@home:/dir

  16. Through a firewall tar cf - . | ssh user@firewall ‘ssh fileserver “cd /dir ; tar xvf -”’

  17. The other way ssh user@home cd /dir ssh user@beamline “cd /data ; tar cf - .” | tar xvf –

  18. The other way, faster ssh user@beamline “cd /dir ; tar cf - . | compress” | uncompress | tar xvf –

  19. The easy other way ssh user@beamline “download yourname” | tar xvf –

  20. What if it stops?

  21. What if it stops? ssh user@beamline “download yourname frame_046.img” | tar xvf –

  22. What if it stops? http://bl831.als.lbl.gov/~jamesh/elves/scripts/sendhome sendhome frame_046.img user@home:/dir

  23. What if I’m confused?

  24. What if I’m confused? 4 steps

  25. 1) Generate a filename list find .-name ‘*.img’ >! here.log

  26. 2) Generate a “done” list ssh user@home “cd /dir ; find .” >! there.log

  27. 3) Make the “todo” list diffthere.loghere.log | awk ‘/^>/{print $2}’ >! 2bsent.log

  28. 4) Feed “still left” list to tar cat 2bsent.log | tar cTf-- | ssh user@home “cd /dir ; tar xvf -”

  29. What if I can’t login? ssh user@home warning: Connecting to home failed: Connection Refused

  30. “SneakerNet”

  31. “SneakerNet” Tape Firewire DVD

  32. FireWire

  33. Realtime Firewire 1)plug in

  34. Realtime Firewire • plug in • share it

  35. Realtime Firewire • plug in • share it • smbclient

  36. Realtime Firewire tar cf - . | smbclient //laptop/share –U user%password" -N -Tx -

  37. Realtime Firewire find . | tar cTf - - | smbclient //laptop/share –U user%password" -N -Tx -

  38. The easy way http://bl831.als.lbl.gov/~jamesh/beamline/backup_firewire.com.txt backup_firewire.com .

  39. DVD

  40. DVD

  41. HOWTO make a DVD mkisofs -r -J . >! image.iso

  42. HOWTO make a DVD mkisofs -r -J . >! image.iso dvdrecord dev=0,0,0 –data image.iso

  43. The easy way http://bl831.als.lbl.gov/~jamesh/beamline/backup_local.com.txt backup_local.com .

  44. The easy way http://bl831.als.lbl.gov/~jamesh/beamline/backup_local.com.txt backup_local.com . http://bl831.als.lbl.gov/~jamesh/beamline/autolabel.com.txt autolabel.com

  45. DVD data archive

  46. DVD data archive

  47. DVD data archive

  48. DVD data archive

  49. DVD data archive

More Related