GLAST Large Area Telescope: A Fusion of HEP and Astro Computing Richard Dubois

1. GLAST Large Area Telescope: A Fusion of HEP and Astro Computing Richard Dubois Stanford Linear Accelerator Center richard@slac.stanford.edu

2. Outline • Introduction to GLAST & LAT • A HEP detector in space • Code Reuse (Beg, Borrow, Steal) • Bulk Processing: turning around a downlink in an hour • Data Access: Catalogues and Portals • Data and Service Challenges • Astrophysics Analysis • Summary

3. Large Area Telescope (LAT) GLAST Burst Monitor (GBM) GLAST Key Features • Huge field of view • LAT: 20% of the sky at any instant; in sky survey mode, expose all parts of sky for ~30 minutes every 3 hours. GBM: whole unocculted sky at any time. • Huge energy range, including band 10 GeV - 100 GeV • Will transform the HE gamma-ray catalog: • by > order of magnitude in # point sources • spatially extended sources • sub-arcmin localizations (source-dependent) Two GLAST instruments: LAT: 20 MeV – >300 GeV GBM: 10 keV – 25 MeV Launch: Apr 2008. Cape Kennedy 565 km, circular orbit 5-year mission (10-year goal) spacecraft partner: General Dynamics

4. GLAST MISSION ELEMENTS GPS Large Area Telescope & GBM GLAST Spacecraft Telemetry 1 kbps TDRSS SN S & Ku DELTA 7920H GN White Sands LAT Instrument Science Operations Center Schedules Mission Operations Center (MOC) GLAST Science Support Center HEASARC GSFC Schedules GRB Coordinates Network GBM Instrument Operations Center Alerts Data, Command Loads

5.  e– e+ Overview of LAT • Precision Si-strip Tracker (TKR) 18 XY tracking planes. Single-sided silicon strip detectors (228 mm pitch) Measure the photon direction; gamma ID. • Hodoscopic CsI Calorimeter (CAL) Array of 1536 CsI(Tl) crystals in 8 layers. Measure the photon energy; image the shower. • Segmented Anticoincidence Detector (ACD) 89 plastic scintillator tiles. Reject background of charged cosmic rays; segmentation removes self-veto effects at high energy. • Electronics System Includes flexible, robust hardware trigger and software filters. Tracker ACD [surrounds 4x4 array of TKR towers] Calorimeter Systems work together to identify and measure the flux of cosmic gamma rays with energy 20 MeV - >300 GeV.

6. Integrated Observatory in Phoenix, AZ

7. 1 Gev Gamma Incident Gamma e+ e- Note energy flow in direction of incident Gamma Radiated Gammas ~8.5 Radiation Lengths Fusion of HEP & Astro Computing “Science Tools” • Collection of tools for detection and characterization of gamma-ray sources (point sources and extended sources) • source finding • max likelihood fitting (binned/unbinned) • parameterized instrument response • exposure maps • comparisons to model (observation sim) • GRBs, periodicity searches, light curves • Science Tools are FITS/FTOOLS based • for dissemination to astro community • Data distributed to public by Goddard Event Interpretation + full code development environment on linux, windows (mac imminent), code and data distribution, automated code builds, documentation etc etc Full simulation/reconstruction of 1 GeV gamma

8. Instrument Design Considerations  e– e+ Background rejection requirements drive the ACD design (and influence the calorimeter and tracker layouts). Effective area and PSF requirements drive the converter thicknesses and layout. PSF requirements also drive the sensor performance, layer spacings, and drive the design of the mechanical supports. Field of view sets the aspect ratio (height/width) Energy range and energy resolution requirements bound the thickness of calorimeter Time accuracy provided by electronics and intrinsic resolution of the sensors. Electronics • Important Design Considerations: Optimized via simulations - Spot Checked in particle beam tests • Lateral dimension < 1.8 m • Restricts geometric area => FOV • Mass < 3000 kg • Primarily restricts total depth of the Cal • Power Budget 650 W • Primarily restricts number of Tracker channels • Background rejection: • Filter out 97% of downlink on the ground • Use Classification Trees • Effects of Trigger & Onboard Filtering • Hardware trigger scheme • CPU cycle requirements and throughput • data volume per event • Segmentation of ACD • Relative importance and size of side tiles • Rejection efficiency due to gaps and screws

9. Event Processing Flow Root: object I/O needed for structured data with cross linkages • event based processing • C++ framework provides base class definition & services • completely configurable - code loaded at run time when needed

10. Sim/Recon Toolkit

11. Data Challenges • A progression of data challenges. • DC1 in 2004. 1 simulated week all-sky survey simulation. • find the sources, including GRBs • a few physics surprises • DC2 in 2006, completed in June. • 55 simulated days (1 orbit precession period) of all-sky survey. • First generation of LAT source catalogue • Added source variability (AGN flares, pulsars). lightcurves and spectral studies. correlations with other wavelengths. add GBM. study detection algorithms. benchmark data processing/volumes/reliability. • 200k batch jobs - worked out reliability issues (< 0.1% failure rate now) Data challenges provided excellent testbeds for science analysis software. Full observation, instrument, and data processing simulation. Team uses data and tools to find the science. “Truth” revealed at the end.

12. Post DC: Service Challenge • No longer need blind science exercises! • Coordinate simulation studies for science and Operations • a common set of simulations plus a near-constant stream of simulations to support special studies. Develop capabilities outside SLAC as needed using collaboration resources. • Operations • Simulating first 16 orbits of L&EO • Run them through full LAT ground processing chain • Develop shift procedures and train collaborators • Science • Full simulation of 1 orbit-year • Definitive pre-launch dataset for working groups • Expect to require 400 CPU-months to create

13. Service Challenge Eye Candy Pointing with two targets Offline Sim of Onboard GRB Filter GRB Trigger Time 0.02089589834 FirstRA 151.2563276 FirstDEC -38.92002236 First Estimated Error 0.5743404438 nPhot w/ [0,100) MeV 20 nPhot w/ [100,1000) MeV 2 nPhot w/ [1,10) GeV 0 nPhot w/ > 10 GeV 0 Trigger window size 40 EnergyCut -1 Alert notice! LSI +61 303

14. Pipeline Processing • Started with STScI’s OPUS - then rolled our own • Features: • execute independent tasks • keep track of state in db • web view/admin of jobs • use dataset catalogue (db) to track files • expect millions of files! • Java/Tomcat, jsp - not GLAST specific

15. The Hardest Task: Downlink Processing Calibration Process each downlink before the next arrives: ~100 cores for 1.5 hrs Monitoring On success: put Humpty back together again Verify Register Merge Clean Merge Digitization Do monitoring Reconstruction Split input data into ~100 parallel pieces

16. Automated Source Monitoring

17. Usage Plots: Activity Summary Many details stored per step in oracle: web displays to track usage and performance

18. Data Portal/Catalog Access/ Authentification handled by web Events, file size, run range automatically set by “crawler” Browsable tree of datasets Supports mirroring at multiple sites Meta-data added by creator

19. Skimmer: Data to the user Access to data will require registration with GLAST member db • Can skim any data from catalog • Data available as root and/or fits files • Skimmer jobs parallelized using Pipeline • Need xrootd to spread disk load, avoiding individual disk server overload • Output available for download for 10 days

20. Computing Resource Projections Providing resources for: flight data, reprocessing, simulations, user analysis Currently: 350 TB disk & 400 cores Add 250 & 400 for 2008

21. xrootd • Beginning to use xrootd • System developed at SLAC for BABAR to manage large datasets • Distributes files across disks • Maximizes throughput • Minimizes manual disk management • Automates archiving datasets to (and restoring from) tape • Provides more reliability and scalability than NFS • Supports access control based on GLAST collaborator list query redirector File servers STK tape silo

22. Conforming to HEASARC FTOOLS • Agreed from the beginning with Mission that science tools would be jointly developed with (and distributed by) Science Support Center and adhere to FTOOLS standard • Atomic toolkit with FITS files as input/output to a string of applications, controlled by IRAF parameter files • Use scripting language to glue apps together • Very different from the instrument sim/reconstruction code! • Shared code development environment, languages • Caused a certain amount of early tension, having to bifurcate coding styles. People are spanning both worlds now. Select events Create Exposure Map Compute Diffuse Response Do Max Likelihood Fit

23. Gamma Ray Analysis: Model Fitting • A scarcity of photons in the GeV range… :-( • Must do max likelihood model fitting • Use parametrized instrument response functions for energy, angular resolution and effective area • Tabulated exposures • Computationally intensive for crowded regions of sky • HEP approach would be to perform full simulations of the sky using complete knowledge, including correlations, of the instrument performance • It remains to be seen in practice whether this approach is needed or feasible • Note that a recent 2 month orbit full simulation took ~500 CPU-days to perform • BUT - that was one elapsed day on the batch farm

24. Summary • GLAST Observatory approaching final testing now • Launch in early 2008 • LAT use HEP techniques to handle science data stream and produce photon list • HEASARC FTOOLS for mainstream astrophysics analysis • It remains to be seen whether HEP’s extensive use of simulations will extend into the data taking era • Invaluable pre-launch • Will “error is in the exponent” make the extra analysis precision unnecessary?