Canadian Light Source

1. Canadian Light Source Elder Matias (CLS) EPICS April 09 Collaboration Meeting

2. Index • Where is the CLS? • CLS – Current projects • ScienceStudio • ANISE – Data streaming expansion to ScienceStudio

3. Where is Saskatoon?

4. 1964 Saskatchewan Accelerator Lab (SAL) was established for chemistry and nuclear physics research.

5. 1964 Saskatchewan Accelerator Lab (SAL) was established for chemistry and nuclear physics research. • 1999 SAL operations were discontinued, and the accelerator was refurbished as an injector for a new storage ring. • SR1 Machine Parameters • 2.9 GeV DBA lattice with 12-fold period • 170.88 m

6. Phase 1 (operation): Total of 7 scientific and 2 diagnostics beamlines. Phase 2 (commissioning): Total of 7 additional beamlines and building expansion for medical imaging. Phase 3 (pre-design): Total of 8 additional beamlines and building expansion.

7. Major Projects • Common • Control Server virtualization • Accelerator • Linac upgrade - Replace/upgrade Linac Modulators, RF Sections, Power Supplies • Transverse feedback and bunch cleaning • Upgrade to existing power supplies • Injection with shutters open • Beamlines • Phase 2 Beamlines – Final commissioning • Phase 3 Beamline Projects including a building expansion • ScienceStudio

8. vmWare ESX Virtualization

9. ScienceStudio Elder Matias (CLS) EPICS April 09 Collaboration Meeting

10. What Is Web 2.0? • In plain English …. • Automating tedious tasks using web technology • Tools to help people and software collaborate

11. Scientific American May 2008 Science 2.0 – The Risk and Reward of Web-Based Research --------------------------------- “Our real mission isn’t to publish journals but to facilitate scientific communication” Timo Hannay – Head of Web Publishing at Nature Publishing Group

12. Traditional Beamline Controls • EPICS Tools • Configuration Tool • User GUI & ROOT

13. User Access to Synchrotrons • What middleware was needed to resolve the limitations? • Workflow management Engine for the User Office • Web Portal for remote data access (during and post experiment) • Enterprise Service Bus and SOA to integrate internal and external data analysis services • How do your plans meet the needs • Users will have frequent remote access to the VESPERS beamline at the Canadian Light Source under conditions where many collaborators can participate in the experiment.

14. Science Studio serves three purposes: • Management of all aspects of a scientific experiment including data storage, collaboration with others, processing of data; • Control of, or interaction with, remote experiments on the CLSI VESPERS Beamline and UWO Nanofabrication Laboratory and • User Services (sample management, scheduling, peer review, user training)

15. Team: People and Orgs • System Requirements • Testing • Data Analysis/Grid Computing • Remote Control • User Services • System Deployment • Integration • User Office Software • Scientific Workflow Engines • System Architecture

16. Team: People and Orgs Mike Bauer Stewart McIntyre Marina Suominen Fuller Jinhui Qin Nathaniel Sherry Dionisio Medrano Dylan Maxwell Daron Chabot Elder Matias Yuhong Yan Zahid Anwar Ludeng (Eric) Zhao Dan Ni Yaofeng Xu Chris Armstrong John Haley

17. Web Application Beamline Control Module VESPERS HTTP JMS CA SAN DB System Architecture • VESPERS Beamline • EPICS control system • Beamline Control Module (BCM) • Web Application • Database • File Storage • Web Interface

18. Web Application Beamline Control Module VESPERS HTTP JMS CA SAN DB VESPERS Beamline • VESPERS — Very Sensitive Elemental and Structural Probe Employing Radiation from a Synchrotron • A bending magnet beamline on sector 6 at the Canadian Light Source synchrotron in Saskatoon, Saskatchewan. • A hard x-ray microprobe with an energy range of 6 to 30keV. • Techniques: X-Ray Fluorescence (XRF) & X-Ray Diffraction (XRD)

19. VESPERS Endstation CCD Detector (XRD) Microscope Sample MCA Detector (XRF)

20. Web Application Beamline Control Module VESPERS HTTP JMS CA SAN DB EPICS Low-level Control System • EPICS — Experimental Physics and Industrial Control System • The standard control system at the CLS. • EPICS consists of a network of Input-Output Controls (IOCs) which are connected to directly to devices. • An IOC provides many Process Variables (PVs) which relate to either an input or output from a device and have a unique name. • Channel Access (CA) is used to read or write to any PV without knowing which IOC provides the PV. • More than 50,000 PVs in the CLS control system.

21. EPICS at the CLS

22. Web Application Beamline Control Module VESPERS HTTP JMS CA SAN DB Beamline Control Module (BCM) • The BCM provides a high-level interface to the low-level control system (EPICS). • Logical and physical separation of business logic and control logic. • Virtual device abstraction that provides independence from low-level control system. • Virtual devices can be logically organized into a device hierarchy. • Basic devices can be combined to build more functional devices. • Communication with external applications using two message queues (ActiveMQ).

23. Web Application Beamline Control Module VESPERS HTTP JMS CA SAN DB Web Application • A J2EE Servlet application that provides a web-based interface Science Studio. • Tools: Spring (MVC), iBATIS (ORM), JSecurity (Apache Ki), Apache Tomcat • Divided into two parts: the Core application and the VESPERS beamline application. • Core application is responsible for providing access to the business objects. • VESPERS application is responsible for remote control of the VESPERS beamline.

24. Web Application Beamline Control Module VESPERS HTTP JMS CA SAN DB Database • Metadata associated with the operation of a remote controlled beamline and the organization of experimental data collected on that beamline. • A project is the top level organizational unit and is associated with a project team. • A session defines a period of time allocated to a project team to conduct experiments. • An experiment relates a sample and the technique being applied to that sample. • A scan records the location of the acquired experimental data.

25. person facility laboratory instrument technique project_person project Instrument_technque project_role sample session experiment scan Database Schema

26. Web Application Beamline Control Module VESPERS HTTP JMS CA SAN DB Experimental Data Storage • Experimental data is stored at the CLS. • Common directory structure shared with other beamlines. • A large data storage facility is now operational at the University of Saskatchewan as part of WestGrid.

27. Web Application Beamline Control Module VESPERS HTTP JMS CA SAN DB VESPERS Web Interface • Rich web interface to Science Studio and the VESPERS beamline. • Designed to be used over commodity broadband internet. • Developed for the Firefox web browser without any additional plugins or extensions. • Known to work with other browsers, but requires the Canvas HTML tag. • AJAX is used for the VESPERS interface to provide device values in pseudo real time. • ExtJS, a JavaScript framework, provides many advanced GUI elements.

28. Beamline Setup

29. Experiment Setup

30. XRF (X-Ray Fluorescence)

31. Beamline Hutch Cameras

32. Experimental Data Viewer

33. User Office Workflow CLS health & safety inspection 6-month cycle • Mar • cientist packs sample Many other tasks • Perform Experiment • Return Sample • Take Survey • … CLS call for proposals CLS grants scientist Beamline time CLS gathers proposals I wonder if CLS received my sample yet? Scientist must complete Online SS training CLS reviews proposals Proposal submission To CLS Goal: Many tasks in proposal & sample management at CLS To develop a workflow management system that • manages ordering of tasks e.g. (training beforeshipping) • Tracks manual as well as SS task progression

34. User office Workflow Status • Workflow Management Engine • Beamline User • User Office Features Open source Petri-nets based Direct support for workflow control flow patterns Ability to interact with web services declared in WSDL Relies on XML standards e.g. XPath and XQuery for data & doesn’t use proprietary languages Architecture System Core: YAWL engine. Engine instantiates specifications designed using YAWL designer. managed by the YAWL repository Environment composed of YAWL services inspired by “web services” paradigm, end-users, applications, and organizations are all services in YAWL. • Task :Training • Completed • Notify • Record Progress • Approved • Notify

35. Screenshot: User Training Test Creation

36. Screenshot: User Survey Taking Page

37. Screenshot: User Survey Edit Page

38. Screenshot: Workflow Sample Management

39. Screenshot: Workflow Call for Proposals

40. User Office Workflow Example • Prototype Implementation 1. CLS issues a call for proposals and gives deadline2. Beamline users submit proposals3. User Office administrator ends registration or extends deadline4. User Office administrator assigns proposals to user office reviewers5. Reviewers look at proposals and rank them6. User Office looks at ranking and chooses the proposals to accept7. Accepted proposals contact persons are notified8. Beamline User completes training (web service)9. After training is completed (simulated by a delay) the CLS is notified

41. Scheduling Module • Goal: To automate the review process and the method by which beam time is allocated and scheduled to users depending on • the access mechanism chosen by the user and • the stage of operation (construction, commissioning or operation) of the beamline. • Side effects: • Facilitate the management of cycles, runs and modes of operation • Use automatic scheduling to handle more scheduling conditions and constraints than human beings are able to handle manually and identify optimal solutions.

42. Scheduling Module Features Users Submit proposals INPUT: CONSTRAINTS 1. One beamline per experiment 2. Start time after release time 3. Only eligible beamlines can be selected . . 7. No overlap of experiment per beamline SEARCH AND CONSTRAINT SATISFIABILITY: Integer Programming and Heuristic Algorithm Schedule OUTPUT:

43. S: Kα • Cr: Kα & Cr: Kβ • Fe: Kα & Fe: Kβ • Ni: Kα & Ni: Kβ • X-Ray Fluorescence (XRF): Reveals Elemental Composition • Characteristic Element Lines Selected and Mapped Over a 2D Scan Area • 2D Maps Generated for Selected Elemental Lines

44. Peak Search • X-Ray Diffraction (XRD): Reveals Structural Information • Peak Fitting and Indexing of Image Set to Create a Grain Orientation Map Indexing Process • Grain Orientations • New C Programme – Matched Peak • New C Programme – Expected Peak • Old IDL Programme – Matched Peak • Old IDL Programme – Matched Peak • The XRD Indexing programme examines the locations of peaks in an image in order to determine the kind of lattice structure the samples constituent atoms are arranged in. Shown here are the results of an older indexing programme written in IDL, and the new indexing programme, written in C. The new indexing programme is proving to be more versatile, and more reliable than the old programme, often indexing sets of data that the old programme failed with.

45. How do I get access to a HPC Machine? • Compute Canada • Responsible for High Performance Computing in Canada • Each regional grid is a member of Compute Canada • ACEnet – Atlantic Canada • CLUMEQ - Quebec • SCINET - UofT • HPCVL – Queens, Royal Military CollageSt. Lawrence, Carlson, Ottawa, … • RQCHP - Quebec • SHARCNET - Ontario • WESTGRID – Western Canada

46. Grid Data Storage? • UofS is the host for the new WestGrid data storagefacility • Cost: $3.2 M • Includes on-line andarchival storage • Two sites on campus • Photo: tape backup unitholding 6,000 tape(each @1TB)

47. IBM Cell Processor (3.2 GHz)

49. ANISE Elder Matias CLS – 09-04-28

50. ANISE: Active Network for Information from Synchrotron Experiments “Active” means near-instantaneous stream processing of complex data during transfer to the user or to storage. Cell processing using Infosphere Streams software from IBM and lightpath provided by CANARIE network. Distributed processing on facilities provided by SHARCNET and WESTGRID. Objective: Develop such a network to provide processed results from experiments such as Laue diffraction at APS (34-ID) and VESPERS at CLS • The network would assist the integration of diffraction data from multiple and large area detectors. • The network would facilitate faster resolution of research problems and free up time for more users. • The network would encouage common data formats and protocols leding to closer collaboration.