1 / 38

Cindy Zheng For PRAGMA Grid/Cloud Team And the PRAGMA communities

PRIME 2011 Tutorial PRAGMA Grid/Cloud Access. Cindy Zheng For PRAGMA Grid/Cloud Team And the PRAGMA communities Pacific Rim Application and Grid Middleware Assembly http://www.pragma-grid.net http://goc.pragma-grid.net. Grid and Cloud. What are they? Collection of resources for sharing

shauna
Download Presentation

Cindy Zheng For PRAGMA Grid/Cloud Team And the PRAGMA communities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PRIME 2011 Tutorial PRAGMA Grid/Cloud Access Cindy Zheng For PRAGMA Grid/Cloud Team And the PRAGMA communities Pacific Rim Application and Grid Middleware Assembly http://www.pragma-grid.net http://goc.pragma-grid.net

  2. Grid and Cloud • What are they? • Collection of resourcesfor sharing • Differences • Grid • Resources glued by agreed and verifiable trust • Certification • GLOBUS • Provide system/software environment for some applications and users • Cloud • Users setup applications on systems • Cloud provide mechanisms to allow users run their systems as virtual machine (VM) in shared resources

  3. Grid Trust • Organizations and policies • International Grid Trust Federation (IGTF) • APGridPMA • EUGridPMA • TAGPMA • Accreditation • Software • Certificate Authorities (CA) • Set up CA services, policies • Issue/manage certificates (user, server, services, …) • GLOBUS • Identify users and resources in a cohesive fashion • Map user certificate to a local account • Interface local job managers

  4. PRAGMA CAs • PRAGMA-UCSD CA (https://goc.pragma-grid.net/ca) • Accredited by APGrid PMA • Included in IGTF distribution • Only issue a certificate when needed • PRAGMA Experimental CA (http://rocks-56.sdsc.edu/exp-ca) • Only accepted by PRAGMA grid site • PRIME 2011 students will use PRAGMA experimental CA unless your project involved with other grids

  5. Grids/Clouds and PRAGMA Grid/Cloud • Grids/Clouds • Most are centrally funded • PRAGMA grid/Cloud is unique • Grass-root • Voluntary contribution • Open (PRAGMA member or not, pacific rim or not) • Long-term collaborative working experiment • Heterogeneous • Funding • No uniform infrastructure management • Open to all sciences and applications • All varieties of site policies, system and network environments • Friendly, helpful, close relationships • More challenging and rewarding • Good for development, collaborations, integrations and testing

  6. PRAGMA Grid/Cloud UZH Switzerland UZH Switzerland JLU China AIST OsakaU UTsukuba Japan CNIC China KISTI KMU Korea IndianaU USA SDSC USA LZU China LZU China ASGC NCHC Taiwan HKU HongKong CICESE UNAM Mexico UoHyd India ASTI Philippines NECTEC KU Thailand CeNAT-ITCR Costa Rica HCMUT HUT IOIT-Hanoi IOIT-HCM Vietnam UValle Columbia MIMOS USM Malaysia IHPC/NGO NTU Singapore UChile Chile MU Australia BESTGrid New Zealand 26 institutions in 16 countries/regions,22 compute sites, 9VM sites(+ 6 site in preparation)

  7. PRAGMA Grid Software Layershttp://goc.pragma-grid.net/pragma-doc/userguide/join.html Applications FMO Savannah MM5 CSTFT Siesta AMBER Phylogenetic … Application Middleware Infrastructure Middleware Ninf-G Nimrod/G Mpich-GX … Gfarm SCMSWeb CSF MOGAS … Globus (required) Local job scheduler (require one) SGE PBS LSF SQMS …

  8. PRAGMA Grid/Cloud Compute Resourceshttp://goc.pragma-grid.net/pragma-doc/computegrid.html

  9. PRAGMA Grid/Cloud Members and Teamhttp://goc.pragma-grid.net/wiki/index.php/Site_status_and_tasks • Team members • >240 and growing • one management contact / site • 1~3 technical support contact / site • 1~4 application drivers / application • 1~5 members / Middleware development team • Experts • Application (various scientific domains) • Middleware (various grid middleware) • Security (grid security) • … • Communications • Email • Mailing list – pragma-grid-team@googlegroups.com • Coordinator – zhengc@sdsc.edu

  10. PRAGMA Cloud Access • Cloud computing is new • Access procedure has not been formalized • Contact Cindy for detail info and arrangements • The rest of the slides shows how to access PRAGMA grid

  11. Obtain PRAGMA Grid Access • Find guide document at http://goc.pragma-grid.net/wiki/images/5/51/PRIME-accounts.doc • Be aware • When cut/paste a command line • Microsoft word may change • A hyphen to a line character • Double-hyphen to a long line character • Learn UNIX concepts and commands • You need for your project • Know how • Understanding (Know why) • Account application as an example • Learn more online, google http://mally.stanford.edu/~sr/computing/basic-unix.html • Try them, hands-on • But be careful with “rm” (remove) command!

  12. Same Basic UNIX Commands • List files $ ls $ ls -l (show file permission, ownership, etc.) $ ls -a (show all files, including hidden files) • Change directory $ cd (go to my home directory) $ cd ~/.ssh (go to .ssh subdirectory under my home directory) • Copy files locally $ cp <local-source-file> <local-destination-path> • Copy files between local and remote systems use SSH key $ scp <local-source-file> <remote-host-name>:<remote-path> $ scp <remote-host-name>:<remote-path> <local-destination-path> • Remember the command, but forgot how to use it $ <command> --help $ man <command>

  13. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  14. Need an UNIX System on Laptop (Step 1.1) • UNIX on laptop • MAC OS 10 • HD • Applications • Utilities • Terminal • Windows • Install Cygwin • Click • LINUX • Login

  15. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  16. Create SSH Key(Step 1.2) • Secure remote login • Generate key files $ ssh-keygen -t rsa • Use default path for ssh key files • Good passphrase (hard to guess, no dictionary words) • Remember the passphrase • View SSH key files (private vs. public) • From Cygwin $ ls ~/.ssh -r--------@ 1 cindyzheng staff 951 Mar 19 01:10 id_rsa -rw-r--r--@ 1 cindyzheng staff 234 Mar 19 01:10 id_rsa.pub -rw-r--r--@ 1 cindyzheng staff 8414 Mar 19 01:10 known_hosts • From Windows c:\Program files\Cygwin\Home\<Your-user-name-on-laptop>\.ssh • Backup SSH key files • USB drive, take with you • Can be used thereafter • Only on secure systems • Possibly compromised key • Immediately inform all public key holders (system administrators)

  17. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  18. Account Application(Step 2) • Email to Cindy • Your full name • Institution name: UCSD • Address of the institution: UCSD • Country of the institution: USA • Your phone number • Your fax number <optional> • Your email address • The purpose of using PRAGMA grid: PRIME • Your/your institution's contribution to PRAGMA (can be work, resources): work • Prefered login names (1st, 2nd, 3rd, but may or may not be granted) • Prefered UNIX shell (may or may not be granted) <optional> • Hostname(s) and IP address(es) of the system(s) which you launch your applications (This is required for PRAGMA grid site systems to open their firewalls in order to allow you access.) <ask your foreign advisor for it> • Your ssh PUBLIC key (NEVER and NEVER give anyone your private key file!! Nor your ssh password!!)

  19. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  20. Cindy • Create user info pack • Create an UNIX user account on rocks-200.sdsc.edu • Email user • Account name on rocks-200.sdsc.edu • Certificate request license id

  21. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  22. Test SSH key(Step 3) • Login to rocks-200.sdsc.edu $ ssh <username>@rocks-200.sdsc.edu • Enter passphrase Last login: Tue Apr 5 11:37:56 2011 from dyn137-110-115-232.ucsd.edu Rocks 5.3 (Rolled Tacos) Profile built 04:25 15-Apr-2010 Kickstarted 22:21 14-Apr-2010 • In case ssh fails $ ssh –v <username>@rocks-200.sdsc.edu Email output to Cindy • First time login on rocks system, ssh-keygen ran automatically • Accept default ssh file path • Empty password • Only used internally • not recommended elsewhere

  23. Make a User Certificate Request(Step 3) $ cd $ /opt/pragma-ca/bin/grid-certreq -sv ra.pragma-grid.net:pragma-exp_ra -new <license_id> -g "SMIME user" -sou ---------(Sample output)------------- creating a certificate signing request ------------------------------------------- generate private key (size 1024 bit) ...................................................oo ....................oo ------- input user subject information -------- email can be omitted by putting a char of '.’ input ou : PRIME  type exactly as this input user name : Cindy Zheng  input your full name input user email : cindy@sdsc.edu  input your email address, must! ------- please confirm your inputs ------------ GROUP : SMIME user SUBJECT : CN=Cindy Zheng, Email=cindy@sdsc.edu do you continue operation? (yes/no/retry)[y]:  press return trying to connect RA server : ra.pragma-grid.net (11412) ... ok. request for issuing a new certificate ... ok. your request is accepted. (AcceptID=0000003) <- remember it CA operator will send an email to tell a result. save a CA certificate file : /home/cindy/.globus/cacert.pem <- note cert request files location save a private key file : /home/cindy/.globus/userkey.pem <- note cert request files location Input PASS Phrase:  type a good password Verifying - Input PASS Phrase:  retype the password (remember it!) • Email Cindy the acceptID

  24. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  25. Cindy • Issue user certificate • Off-line CA server • Inform user via email

  26. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  27. Retrieve User Certificate(Step 4) • Login to rocks-200.sdsc.edu $ ssh<username>@rocks-200.sdsc.edu • When prompt, enter your SSH password • Retrieve user certificate $ /opt/pragma-ca/bin/grid-certreq -sv ra.pragma-grid.net:pragma-exp_ra -em<your-email> -recv<your-acceptID> trying to connect RA server : ra.pragma-grid.net (11412) request for exporting a certificate ... ok save a CA certificate file : /home/cindy/.globus/cacert.pem save a certificate file : /home/cindy/.globus/usercert.pem • View and understand certificate files $ ls -l .globus -rw-r--r-- 1 cindyzheng staff 2878 Mar 23 18:26 cacert.pem -rw-r--r-- 1 cindyzheng staff 4998 Mar 23 18:26 usercert.pem -r-------- 1 cindyzheng staff 1743 Mar 23 18:26 userkey.pem

  28. Backup Certificate Files(Step 4) • Copy .globus directory from rocks-200.sdsc.edu to your laptop • Start Cygwin/terminal on your laptop (Should land in your home directory) • $ scp-r<username>@rocks-200.sdsc.edu:/export/home/<username>/.globus . • Copy c:\Program files\Cygwin\Home\<Your-user-name-on-laptop>\.globus to your USB backup disk • Take with you anywhere • Can be used usually 1 year • Only on secure systems • Possibly compromised certificate • Immediately inform certificate issuer (Cindy)

  29. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  30. Test Globus Authentication(Step 5) On rocks-200.sdsc.edu: • Create a proxy $ grid-proxy-init Your identity: /O=grid/O=pragma/OU=PRIME/CN=Cindy Zheng Enter GRID pass phrase for this identity: <- input passphrase Creating proxy .......................................................... Done Your proxy is valid until: Thu Apr 7 02:18:11 2011 • Test globus authentication $ globusrun -a -r rocks-200.sdsc.edu GRAM Authentication test successful • Email Cindy if successful, otherwise email output to Cindy • To create longer proxy, specify hours $ grid-proxy-init –hours 100

  31. Test Job Submission(Step 5) $ globus-job-run rocks-200.sdsc.edu/jobmanager-sge /bin/date /opt/gridengine/default/spool/rocks-65/active_jobs/330008.1/pe_hostfile compute-0-12 Tue Apr 21 13:06:18 PDT 2009 rm: cannot remove `/tmp/330008.1.all.q/rsh': No such file or directory  ignore this for now Note: depending on how busy the system is, you may need to wait a bit. To find out the status of SGE queue, open/login to another shell and run $ qstat –f queuename qtype used/tot. load_avg arch states ---------------------------------------------------------------------------- … ---------------------------------------------------------------------------- all.q@compute-0-6.local BIP 1/2 0.06 lx26-x86 328085 0.56000 data jas dr 04/15/2009 23:29:01 1 ---------------------------------------------------------------------------- … ##################################################################### - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - ##################################################################### 330010 0.00000 data zhengc qw 04/21/2009 13:18:39 1 Email Cindy if successful, otherwise email output to Cindy

  32. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  33. Cindy • Add your email to pragma-grid-team list • Email pragma-grid-team@googlegroups.com ask site adminitrators to create account on their systems • Site administrator will email your username to you and me after your account is created

  34. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  35. Test Access To Your Host Site(Step 6) • Host system name/ip (where you will be launching your applications from) • Do the same tests as you have done with rocks-200.sdsc.edu • Test SSH (Slide 20) • Test Globus authentication (Slide 29) • Test job submission (Slide 30) • More testing tips • http://goc.pragma-grid.net/wiki/index.php/User_Testing • Google

  36. Test Access to Each Site(Step 6) • Test SSH • Find the target system name at http://goc.pragma-grid.net/wiki/index.php/Site_status_and_tasks • Test SSH from your host system to the target host (should work) • Test SSH from your laptop to the remote host (May work) • Issues about firewall and site policies • Test Globus authentication • From the system at your host site • To the target system • Test job submission • From the system at your host site • To the target system • Problem with any test • Save output • Check tips at • http://goc.pragma-grid.net/wiki/index.php/User_Testing • Google • Email site contact and cc zhengc@sdsc.edu

  37. Process At A Glance PRIME Student Cindy start Step 2 Email application Step 1.2 Create a SSH key Create account on rocks-200 Laptop platform MAC Step 3 Test SSH, Request certificate windows Issue certificate Step 4 Retrieve certificate Step 1.1 Install Cygwin Step 5 Test Globus authentication and job submission confirmation Request to all sites sites reply Step 6 Test SSH, Globus and job submission to each site

  38. Use PRAGMA Grid • Information online goc.pragma-grid.net • member sites (hostname, contacts, user guide, …) • computational grid (resource table, jobmanager, cpu architecture, …) • Applications (as examples) • SCMSWeb (realtime system status, software catalog, etc.) wiki.pragma-grid.net • For Users • Ask questions (email, skype) • Email, skype • Site contact – always cc Cindy • Looking for experts – ask Cindy • If Cindy is not around, pragma-grid-team@googlegroups.com

More Related