Toward Lights-out Operation and Full Hardware Simulation in JBluIce-EPICS

1. Toward Lights-out Operation and Full Hardware Simulationin JBluIce-EPICS Mark Hilgart, Ruslan Sanishvili, Sudhir Pothineni, Sergey Stepanov, Oleg Makarov, Nagarajan Venugopalan, Michael Becker, Craig Ogata and Robert F. Fischetti GM/CA@APS, Advanced Photon Source X-Ray Sciences Division of Argonne National Laboratory USA

2. JBluIce-EPICS: Development plan • GM/CA@APS is a macromolecular crystallography beamline • JBluIce-EPICS is our custom data acquisition software • We have two main development themes: • Flexibility • Hardwareless development • Interactive tutorials • Remote mode with Java Web Start • Automation • Full lights-out capability • Selectively interactive JBluIce performing loop determination Mark Hilgart - JBluIce-EPICS - September 25, 2012

3. Hardware simulation JBluIce-EPICS GUI + Logic • Integrated in to JBluIce, runs in the same JVM • Simple • All part of a single package • “Just works” – low maintenance, no external dependencies • All configuration in one place, in source control • Less realistic • Real servers bring more details into the simulation • But this realism is unnecessary • Virtual hardware is version controlled with software • Old versions can now be run on obsolete hardware • Two parts • Hardware abstraction layer (HAL) for redirecting hardware calls • Simulated devices • Current state • Allows JBluIce to run on any computer, only disk I/O is (for now) needed • Development of most functionality is done without a beamline HAL (EPICS, SQL, Web, Disk, Scripts) (or) Simulated Devices Beamline HAL redirects I/O calls to simulated or real hardware Mark Hilgart - JBluIce-EPICS - September 25, 2012

4. Hardware Abstraction Layer (HAL) JBluIce-EPICS GUI + Logic • All I/O in JBluIce is done through the HAL • Every call to I/O packages was replaced • HAL consists of five top-level classes • EPICS.java: beamline hardware access • SQL.java: configuration and temporary storage • Web.java: HTTP connections to cameras • Disk.java: will provide a virtual disk • ExecTask.java: runs command-line programs • Each top-level class redirects calls to either a hardware or sim class • Importantly, simulation does not clutter non-simulation code • The only change to production code is the addition of the five HAL redirection classes • No other production code even knows about the simulation • Production code is always written for real hardware (only) • If testing shows bugs when running on real hardware, the simulated devices are fixed HAL (EPICS, SQL, Web, Disk, Scripts) (or) Simulated Devices Beamline HAL redirects I/O calls to simulated or real hardware Mark Hilgart - JBluIce-EPICS - September 25, 2012

5. HAL: EPICS Layer EPICS server EPICS server EPICS server • EPICS is a protocol for communication with beamline hardware • All communication is done with Process Variables (PVs) • For example, the position of a motor may be stored in a PV called beamline1:CCD:position • A command to move a motor may be sent in a PV called beamline1:CCD:start • EPICS is a database with callbacks on every variable • Whenever a variable changes, any listening client is notified • EPICS clients receive updates at 10Hz value change instant (<100ms) notification broadcast EPICS client (e.g. JBluIce) EPICS client (e.g. Script) EPICS client (e.g. JBluIce) Standard EPICS servers broadcast events to all clients Mark Hilgart - JBluIce-EPICS - September 25, 2012

6. HAL: EPICS simulation Hardware Simulation • Hardware vs. simulation • Hardware mode HAL simply passes updates between JBluIce and EPICS servers • In sim mode the simulated EPICS server reflects updates back to JBluIce and simulated devices • This method is not as realistic as running a pure Java IOC, but • It is realistic enough for nearly all development • It doesn’t require any socket communication which may be blocked when JBluIce is deployed over the web • It is simple • Note that delays like the 10Hz EPICS polling delay are not simulated • By making sure our code don’t rely on them it can actually be more robust • Timing issues are debugged during testing • Testing can be faster • An advantage is that in simulated mode, PVs can be created on demand • Different PV configurations can be tried out quickly during development • Datatypes are set by their first outgoing call • In the future we plan to link multiple JBluIce instances’ simulations together JBluIce JBluIce HAL HAL HW EPICS Servers Sim EPICS Server Sim Devices The simulated EPICS server reflects updates to JBluIce and sim devices through the HAL Mark Hilgart - JBluIce-EPICS - September 25, 2012

7. HAL: SQL • Why does JBluIce use MySQL? • Beamline configuration • Used in combination with EPICS for communicating large amounts of data which would be inefficient in EPICS alone • SQL is updated, then a single EPICS variable conveys which data has changed • This gets around the problem of MySQL not supporting callbacks • How is SQL passed through the HAL? • Originally, SQL command strings were constructed in each device class • The first simulated SQL server parsed these strings • A simpler method is to call methods like: • SQL.select(DB.User, “raster_results”, columnList, where_clause); • This method makes both JBluIce device code and the simulated server code simpler • Actual SQL statements are generated only when necessary in hardware mode Mark Hilgart - JBluIce-EPICS - September 25, 2012

8. HAL: HTTP, Disk, and Command-line Programs • HTTP • Simulated HTTP server takes URLs and returns streams • Simulated webcams return static images • Soon will pan/zoom/rotate images based on motor positions • Disk • Disk is needed for • Storing file output from command-line scripts and programs • Saving logs • Checking if images have been written • A virtual disk will support these operations and allow viewing of the files • Command-line programs • Includes Perl scripts and third-party programs like DISTL • In simulated mode, these programs are replaced by threads that run the same way simulated devices do, interacting with the rest of the HAL Mark Hilgart - JBluIce-EPICS - September 25, 2012

9. Simulated Devices • Motors • Supports simple motion and abort requests – errors and compound motions are planned • Each motor’s configuration is set in a config class and includes limits, speed, and start position • Frame Processing Engine (FPE) • FPE handles frame timing • Simulated FPE is fairly simple • It mainly has to move one motor and report success since that’s all JBluIce sees • Images are pre-recorded • This means writing a new FPE for a fast detector that hasn’t arrived yet is relatively easy, so development can start much before the new FPE is ready • Scripts • Perl scripts are developed to handle complex hardware interaction by non-JBluIce developers • Perl could be interpreted in Java but to convert them to interact with the simulated EPICS layer is not feasible • Extremely simplified versions of the scripts are run instead and they are enough to develop around • As with the FPE, it just has to look like it worked to JBluIce • Other EPICS devices, e.g. fluorescence detector • These play back recorded data for fixed scenarios • In development: Cameras and robot simulations Mark Hilgart - JBluIce-EPICS - September 25, 2012

10. Lights-out Operation • Goal: The user should interact with data acquisition only when necessary • So we need to support • Zero interaction: all choices are made automatically • Up-front only interaction: Known parameters are entered before starting • Selective interaction: User intervenes only if they see something • Checkpoint interaction: Process stops at pre-determined tasks JBluIce Screening Tab Mount Center Plan Collect Analyze The data acquisition pipeline: All steps must be automated and linked Mark Hilgart - JBluIce-EPICS - September 25, 2012

11. Diffraction-based centering (Auto-raster) • Chooses where to collect data • Centering can be done manually, optically, or by diffraction • Optical centering is reliable for finding the loop • Most difficult part of integration • Requires distilling a lot of crystallographic knowledge into a well-defined ruleset • Needs to handle all kinds of samples • Iterative process • We make improvements to the algorithm • Users tell us when it fails – why one image should have scored higher than another Mount Center Plan Collect Analyze Automatic centering is the most challenging link Mark Hilgart - JBluIce-EPICS - September 25, 2012

12. Auto-raster challenges • Currently there are two main issues • Determining crystal outlines • Weakly diffracting crystals generate only a few diffraction spots • A few spots are actually ideal – more means you are using too much dose • But knowing just how much dose to use is difficult • Also picking out these few spots is challenging • Accurately reporting quality • Simply counting spots is not reliable • Twinning generates more spots but indicates poorer quality • Here we are working on improving the scoring algorithm Mount Center Plan Collect Analyze Automatic centering is the most challenging link Mark Hilgart - JBluIce-EPICS - September 25, 2012

13. Strategy & Analysis • Strategy determines data collection parameters like which angle to collect at • Currently takes ~1 minute which holds up the pipeline • We are adding to our computing cluster to minimize this time • Directly linking strategy to data collection is straightforward • User input can improve data collection • Input can be taken before or during data collection • Analysis tells the user as much as possible about their data as can be done automatically • Will also always be improving because the goal is to help or even automatically generate protein structures JBluIce Strategy Display Mount Center Plan Collect Analyze Analysis always has room for improvement, it can never match human interaction Mark Hilgart - JBluIce-EPICS - September 25, 2012

14. Conclusion • Full hardware simulation • Extra development effort is worth the cost just for the productivity improvement • Development never stops because of computing or hardware issues • New hardware can be developed for before it arrives • Integrated simulation means low maintenance and no external dependencies • Development is incremental, it’s useful as soon as it can start without hardware • Will improve the accessibility of JBluIce by letting anyone try it over the web • May enable improved remote operation • Lights-out operation • Frees users to think about only what is important • Each connection must be usable and configurable as it’s made • Auto-raster algorithms are the most challenging aspect • Reliability and robustness are requirements Mark Hilgart - JBluIce-EPICS - September 25, 2012

15. JBluIce-EPICS JBluIce Developers Mark Hilgart Sudhir Pothineni EPICS Developers Sergey Stepanov Oleg Makarov Design Suggestions and Testing Craig Ogata Ruslan Sanishvili Michael Becker Nagarajan Venugopalan Management Janet Smith Robert Fischetti Mark Hilgart - JBluIce-EPICS - September 25, 2012