IDAaaS – ISIS Data Analysis as a Service: The Story So Far

1. IDAaaS – ISIS Data Analysis as a Service: The Story So Far Russell Ewings – ISIS Excitations Group Leader

2. IDAaaS – ISIS Data Analysis as a Service: The Story So Far • Some context – the Excitations group, bottlenecks to scientific output • The IDAaaS system – what it is, how we use it, how we want to use it • What we have learned • The longer term future

3. Acknowledgements Julimar Romero Frazer Barnsley Warren Jeffs Jeremy Spencer Alex Dibbo Franz Lang Gordon Brown …

4. The name… ISIS Data Analysis as a Service

5. The Excitations group at ISIS

6. The Excitations group at ISIS • Facts and figures: • ~90 experiments per year across 4 instruments • 10 instrument scientists supporting the instruments and conducting own research • Approx. 60% of experiments are done using single crystal samples, 40% on powders (90% : 10% split in time though) • Typical powder experiment: 1 – 2 days, raw data files 0.4GB each (108 pixels), ~100 per experiment, but reduced to ~1GB after processing • Typical single crystal experiment: 4 – 8 days, raw data files same size, ~100 per day (x3 for parallel measurement mode), file size increases post processing! End up with ~200GB per day, ~2TB at experiment end.

7. The Excitations group at ISIS • Scientific context: • Experiments are tightly time-bound – access panels often award only just enough beamtime to do what was proposed, so experimenters need to decide what to measure next very quickly • Users are sometimes neutron scattering experts, but often use it as one of many experimental tools, so skill level at doing experiments and analysing data varies a lot • Broad range of scientific backgrounds

8. Science at a neutron source – how it is supposed to work Proposal to publication route

9. Science at a neutron source – what actually happens Proposal to publication route ??????

10. The Excitations group at ISIS The data bottlenecks • Not big data, but inconvenient data • Typically too large for user sitting in their office with a regular desktop machine (N.B. there are potentially lots of users) • Need to be able to access any bit of the dataset at any time in a highly interactive fashion (i.e. not job-based) • Specialist tools required for visualising and analysing data – need to be carefully curated, can be tricky to get started with them • Need to be able to process data quickly so can look at it on-the-fly during time-limited beamtime • Majority of software used for data analysis and visualisation does not exploit parallel architectures or HPC

11. The Excitations group at ISIS The data bottlenecks • Not big data, but inconvenient data • Typically too large for user sitting in their office with a regular desktop machine (N.B. there are potentially lots of users) • Need to be able to access any bit of the dataset at any time in a highly interactive fashion (i.e. not job-based) • Specialist tools required for visualising and analysing data – need to be carefully curated, can be tricky to get started with it • Need to be able to process data quickly so can look at it on-the-fly during time-limited beamtime • Majority of software used for data analysis and visualisation does not exploit parallel architectures or HPC at all

12. A brief aside about Horace The Ceph file store architecture means you can think about different ways of accessing data to make the processes of generating the large multi-run data file and taking cuts from it significantly faster

13. The IDAaaS system What it is (from the user’s perspective)

14. The IDAaaS system What it is (from the user’s perspective) N.B. Users’ virtual machines are somewhat isolated from one another

15. The IDAaaS system What it is (behind the scenes) • ISIS Data Analysis as a Service • STFC cloud • 200 virtual machines accessible • 7TB RAM • 5000 CPUs in local cloud, ~14000 in wider IRIS cloud • 20 GPUs in local cloud, lots more in IRIS • More of everything being installed pretty much continuously • Web based interface to use-case customised menu of virtual machines • Access to VM desktop either via web browser or remote desktop

16. The IDAaaS system What it is (behind the scenes) • All of the virtual machines access a Ceph file store. This brings advantages in terms of speed of access to data (object vs file storage) • The look and feel of all of the VMs is the same – all built as CentOS 7; the only variations are what software is installed and the machine specs • e.g. • WISH single crystal diffraction uses tools in Mantid that are almost unusable without a GPU and plenty of RAM • Muons just need a compiled version of some software from PSI and access to the (small) raw data files

17. The IDAaaS system What it is (behind the scenes) System monitoring using Grafana (web accessible)

18. The IDAaaS system How we are using it • Currently deployed for 1 group (Excitations) & 1 heavy load crystallography beamline (WISH) • Primary route by which users access, process, view and analyse their data since early 2019 • Usage during beam cycles typically sees between 30 and 40 users logged on at any one time. Declines somewhat during shutdowns to between 10 and 20 active sessions. • Note – above means that being used by a significant fraction of people who are off-site. Expect this group to get steadily bigger

19. The IDAaaS system How we want to use it • Roll out to the rest of ISIS – a further 7 groups, all with variations on their requirements • First 3 new groups (muons, reflectometry, disordered materials) are just starting user acceptance tests • Eventually (in about a year) expect ~300+ users logged on at any one time • Need to take this steadily to ensure sufficient computing resource available, and users’ expectations are successfully met • 24/7 support during beam cycles • HPC link up (e.g. STFC’s SCARF cluster) to be made easier

20. Lessons learned Be really careful about specifying what you want, and make sure that everyone really understands what this means Want to avoid misunderstandings leading to wasted effort!

21. Lessons learned Take care to ensure that the end user does not have to fiddle around with settings once they have logged in Make logging in easy! Usernames and passwords sensible… Think carefully about how legacy files and data are to be dealt with Understand software and hardware dependencies Important to have a triage between end users (and their requests) and the development team so that prioritisation is not just who shouts loudest or gets in first If everything works well, make sure you have enough capacity to cope with demand

22. The future Multi Use Spectrometer for High Rate Observations Of Materials - MUSHROOM • Very high count rates. 50 x highest rate existing instrument for the same energy resolution • 2 str of position sensitive detector coverage • Double elliptic guide to focus on small samples ≤ 1 cm3 • Large dynamic range -1.2 to +20 meV in one shot When we build such instruments, need a realistic plan in place for dealing with the data!

23. Summary • At ISIS we find that the data volumes we produce are inconveniently large, so that a single user with access only to a desktop machine would face severe impediments to analysing their data • The IDAaaS system is up and running for 1 group (Excitations) + 1 instrument (WISH), with more planned in the near future • It allows us to address several of the bottlenecks in the experiment to publication process • Having a stable, well-supported and well-curated set of workflow-specific analysis environments lowers the barrier to entry for new users

25. Spare slides

26. The IDAaaS system How we are using it • During experiment: • Instrument diagnostics (once or twice a day) • Data reduction / conversion (approx. every 5 mins) • Visualisation • (Quick) analysis to decide what to do next

27. The IDAaaS system How we are using it • Post experiment: • More sophisticated visualisation • Detailed corrections (e.g. background features, small sample misalignments) • Finding the most relevant bits of the data • Simulation and fitting • Making plots to go in your paper