180 likes | 456 Views
2. Perceived Problem Grid Usability. If a grid interface or the working environment is (apparently) too difficult to use, it will not be adopted by many of the appropriate usersMost scientists, engineers, and researchers will be (service) end usersCurrently grids are serving power users and do not yet benefit non-computer-technical researchers.
E N D
1. Commodity Workflows for the Grid: An Evaluation of Microsoft Windows Workflow Foundation
Quinn S. Lewis
Grid Computing (COSC6376)
14 December 2006
2. 2 Perceived Problem – Grid Usability If a grid interface or the working environment is (apparently) too difficult to use, it will not be adopted by many of the appropriate users
Most scientists, engineers, and researchers will be (service) end users
Currently grids are serving power users and do not yet benefit non-computer-technical researchers
http://wiki.oucs.ox.ac.uk/esp-grid/Final_recommendations_for_e-Science
http://wiki.oucs.ox.ac.uk/esp-grid/JISC_Proposal_for_Phase_2?action=AttachFile&do=get&target=ESPGRID_miniExtensionProposalAug06v1.pdfhttp://wiki.oucs.ox.ac.uk/esp-grid/Final_recommendations_for_e-Science
http://wiki.oucs.ox.ac.uk/esp-grid/JISC_Proposal_for_Phase_2?action=AttachFile&do=get&target=ESPGRID_miniExtensionProposalAug06v1.pdf
3. 3 One Approach Improve the usability of the tools used to interact with the grid
Should be natural and easy-to-use
Take advantage of other familiar commodity tools and technologies
Windows Workflow Foundation
Free
Platform level component
Used by other (Microsoft) tools
4. 4 Workflows Defined Workflow is a set of activitiesthat describe the implementation of a process
Workflow can be short or long running and can involve people and systems Business Processes and Web Services: Strategies and Standards (John Evdemon)
Keynote.ppt
Images taken from http://msdn2.microsoft.com/en-us/netframework/aa973808.aspx.Business Processes and Web Services: Strategies and Standards (John Evdemon)
Keynote.ppt
Images taken from http://msdn2.microsoft.com/en-us/netframework/aa973808.aspx.
5. 5 Workflows in Technical Computing Accelerating Discovery in Science and Engineering (Fabrizio Gagliardi)
Taken from Santander-Fab.pptAccelerating Discovery in Science and Engineering (Fabrizio Gagliardi)
Taken from Santander-Fab.ppt
6. 6 Pillars for Connected System Business Processes and Web Services: Strategies and Standards (John Evdemon)
Keynote.ppt
In some MS presentations I’ve seen this as “Services & Messaging, Interaction, Workflow, Identity & Accessed, Shared Data”.
Business Processes and Web Services: Strategies and Standards (John Evdemon)
Keynote.ppt
In some MS presentations I’ve seen this as “Services & Messaging, Interaction, Workflow, Identity & Accessed, Shared Data”.
7. 7 Connected Systems Architecture Model Business Processes and Web Services: Strategies and Standards (John Evdemon)
Keynote.ppt
In some MS presentations I’ve seen this as “Services & Messaging, Interaction, Workflow, Identity & Accessed, Shared Data”.
Business Processes and Web Services: Strategies and Standards (John Evdemon)
Keynote.ppt
In some MS presentations I’ve seen this as “Services & Messaging, Interaction, Workflow, Identity & Accessed, Shared Data”.
8. 8 Similarity to OGSA Image taken from https://forge.gridforum.org/sf/go/doc13070;jsessionid=3C9F6C4276EC4D99F021178AA9EB4F71?nav=1Image taken from https://forge.gridforum.org/sf/go/doc13070;jsessionid=3C9F6C4276EC4D99F021178AA9EB4F71?nav=1
9. 9 Workflow Foundation Architecture Reference http://www.informit.com/articles/article.asp?p=418010&seqNum=4&rl=1
Although not used, may be helpful to consider http://www.codeproject.com/dotnet/UnderstandWWF.asp.Reference http://www.informit.com/articles/article.asp?p=418010&seqNum=4&rl=1
Although not used, may be helpful to consider http://www.codeproject.com/dotnet/UnderstandWWF.asp.
10. 10 Work Performed Attempt to use Windows Workflow to connect to grid middleware
WSRF.NET/OGSI.NET
Globus
Condor
Explore re-hosting of Workflow Designer outside of VisualStudio Windows Workflow Foundation: Everything About Re-Hosting the Workflow Designer
http://msdn2.microsoft.com/en-us/library/aa480213.aspx
See also Windows Workflow Foundation Samples
http://msdn2.microsoft.com/en-us/library/ms741723.aspxWindows Workflow Foundation: Everything About Re-Hosting the Workflow Designer
http://msdn2.microsoft.com/en-us/library/aa480213.aspx
See also Windows Workflow Foundation Samples
http://msdn2.microsoft.com/en-us/library/ms741723.aspx
11. 11 WSRF.NET/OGSI.NET From the University of Virginia
Use Microsoft .NET to implement WSRF and OGSI
Takes advantage of .NET attributes
Uses VisualStudio Add-In to generate stubs (client/server) to handle state
Code generation tool didn’t seem to work for web services on ports other than 80
Requires WSE 3.0
12. 12 Condor Exposes job scheduling capability as web services to enable
execution of complex computational workflows
consisting of a number of inter-related jobs
Ported SOAPScheddApiHelper to C#
Reference: http://www.cs.wisc.edu/condor/doc/birdbath.pdf
Reference: http://www.cs.wisc.edu/condor/doc/birdbath.pdf
13. 13 Condor Text to Graphics In code…
TransactionAndStatus tx = schedd.beginTransaction(60);
IntAndStatus clust = schedd.newCluster(tx.transaction);
IntAndStatus job = schedd.newJob(tx.transaction,
clust.integer);
ClassAdStructAndStatus cas = schedd.createJobTemplate(
clust.integer, job.integer,
owner, type, cmd, args, reqs);
Status st = schedd.commitTransaction(tx.transaction);
RequirementsAndStatus reqs_s = schedd.submit(
tx.transaction, clust.integer, job.integer,
cas.classAd);
14. 14 Unsecure Condor Setup Modify condor_config and run condor_restart
ENABLE_SOAP=TRUE
ALLOW_SOAP=*
HOSTALLOW_WRITE=* //or may get “Permission denied” from schedd.beginTransaction(30);
QUEUE_ALL_USERS_TRUSTED=TRUE //see next slide
Collector web services runs on port 9618
http://<machine>:9618 (couldn’t use localhost)
Check CollectorLog for more info
Schedd runs on random port
Check …\log\.schedd_address for port
Start explicitly in condor_config or via collector web service (SCHEDD_ARGS=-p 1134) [Reference] Manage grid resources with Condor Web services
https://www6.software.ibm.com/developerworks/education/gr-wscondor/index.html
Developer APIs to Condor + A Tutorial on Condor’s Web Service Interface
http://www.cs.wisc.edu/condor/CondorWeek2006/presentations/farrellee_tannenba_APIs.ppt
Where to begin?
https://lists.cs.wisc.edu/archive/condor-users/2006-August/msg00322.shtml
Other “magic” condor_config vars
https://lists.cs.wisc.edu/archive/condor-users/2006-September/msg00223.shtmlManage grid resources with Condor Web services
https://www6.software.ibm.com/developerworks/education/gr-wscondor/index.html
Developer APIs to Condor + A Tutorial on Condor’s Web Service Interface
http://www.cs.wisc.edu/condor/CondorWeek2006/presentations/farrellee_tannenba_APIs.ppt
Where to begin?
https://lists.cs.wisc.edu/archive/condor-users/2006-August/msg00322.shtml
Other “magic” condor_config vars
https://lists.cs.wisc.edu/archive/condor-users/2006-September/msg00223.shtml
15. 15 Condor Problem Get "Could not create new cluster" when trying to “IntAndStatus cluster = schedd.newCluster(xact.transaction);”
SchedLog says “QMGT command failed: anonymous user not permitted”
Undocumented answer is to add “QUEUE_ALL_USERS_TRUSTED=TRUE” to condor_config since no security is being used
16. 16 Another Condor Problem Nothing seems to execute but it is submitted and get strange timer errors
ShadowLog
12/9 16:28:47 (24.0) (8000): ERROR: Could not locate valid credential for user 'Quinn@COPERNICUS'
12/9 16:28:47 (24.0) (8000): init_user_ids() failed!
12/9 16:28:47 (24.0) (8000): ERROR: Could not locate valid credential for user 'Quinn@COPERNICUS'
12/9 16:28:47 (24.0) (8000): init_user_ids() failed!
12/9 16:28:47 (24.0) (8000): ERROR "set_user_priv() failed!" at line 517 in file ..\src\condor_c++_util\uids.C
SchedLog
12/9 16:25:43 (pid:4044) my_popen: CreateProcess failed
12/9 16:25:43 (pid:4044) Failed to execute C:\condor/bin/condor_shadow.pvm.exe, ignoring
12/9 16:25:43 (pid:4044) my_popen: CreateProcess failed
12/9 16:25:43 (pid:4044) Failed to execute C:\condor/bin/condor_shadow.std.exe, ignoring
RESOLUTION…. Use condor_store_cred! (Reference)
Don’t use condor_credd if you want something easy
Will need to set a password even if you don’t normally use one (net user Quinn <password>) http://www.kellys-korner-xp.com/win_xp_passwords.htm
17. 17 Globus Can’t consume GT4 web services from .NET
Tried manipulating WSDL
Discovered Microsoft and Globus didn’t use some WS-Addressing mechanism
Thought to use Axis and found out a C++ version exists
Complicated install and language mixing issues
18. 18 Closer to home… UH Work Computational Data Grid for Scientific and Biomedical Applications
Marc Garbey and Victoria Hilford
Rapid application development (use of the Microsoft Visual Studio .NET technology)
Visual modeling interfaces (forms driven Graphical User Interfaces)
Database Connectivity (interface with Microsoft SQL Server 2005)
Query support (clients can store, update, delete, retrieve database metadata)
Context-sensitive, role-based access (Microsoft Windows Server 2003, ASP.NET)
Robust security (HIPPA compliance through Microsoft’s Authentication and Authorization from IIS and ASP.NET)
Connectivity to other biomedical resources (PACS, DICOM, XML)
http://research.microsoft.com/workshops/escience2005/
http://research.microsoft.com/workshops/escience2005/videos/12797/lecture.htmhttp://research.microsoft.com/workshops/escience2005/
http://research.microsoft.com/workshops/escience2005/videos/12797/lecture.htm
19. 19 References http://ws.apache.org/axis/cpp/index.html
http://www.windowsclusters.org/condor_guide.htm
Condor WS not possible w/ 6.8.0 and helper? https://lists.cs.wisc.edu/archive/condor-users/2006-August/msg00350.shtml