XMM data reduction: part II SAS command-line analysis and scripting

1. XMM data reduction: part IISAS command-line analysis and scripting Andy Read

2. Overview • Command-line examples/help • Why use command-line and scripts? • Example scripts : SASmakecalev & SASprepareXMM • Imaging scripts & spectral scripts • Scripts within scripts • Final remarks XMM data reduction: part II

3. Command line examples Setting up the user environment (from Tim’s talk) • To run on XROA system: > sas-setup-6.5.0 initialises software > setenv SAS_ODF (path_to_ODF_directory) e.g. /data/71/tro/xmm_data/0106860101/odf/ > setenv SAS_CCFPATH /data/rb2/sas_ccf > cifbuild fullpath=yes >& cifbuild.log builds ccf.cif– calibration index file > setenv SAS_CCF (path_to_ccf.cif_file) > odfingest odfdir=$SAS_ODF outdir=$SAS_ODF >& odfingest.log builds ***SUM.SAS file in ODF directory – ODF summary file necessary for reprocessing XMM data reduction: part II

4. Command line help - Simple help – parameters and default values XMM data reduction: part II

5. - More - Parameters, parameter types, default values and ‘one-line’ help XMM data reduction: part II

6. - Full help – Links to SAS web pages - Cut and paste link into browser XMM data reduction: part II

7. Why use scripts? • Scripts are just a list of command-line calls/commands run sequentially • There are many powerful command line calls • Can run intensive analysis on many datasets, sources etc. • Can be sure of doing exactly the same thing every time, e.g. running exactly the same procedures on many sources/datasets • No need for interactive use • - can run overnight (no need to be there) • - can run as cron jobs (no need to be even logged in) • Often need to redo entire analysis, e.g. • - New calibration files • - New ODF files • - Mistake, change in e.g. source positions, detection threshold etc. Useful scripting cookbook at http://starlink.rl.ac.uk/star/docs/sc4.htx/sc4.html XMM data reduction: part II

8. Example script : SASmakecalev XMM data reduction: part II

9. Executable t-shell script Comment lines Runs at low priority Number of arguments On-screen help – appears whenever incorrect number of arguments is given (e.g. zero) If correct number of arguments is given (7) then input arguments (1-7) converted to parameters used in script, and we move into the main body of the script XMM data reduction: part II

10. Check that ODF exists (otherwise exit) Get SAS version (for directory naming) Create (if necessary) output directory, and (if allowed) enter XMM data reduction: part II

11. Check ODF Set SAS_ODF environment variable Check for ***SUM.SAS file (Check for ***SUM.ASC file) Create ccf.cif – calibration index file Set SAS_CCF environment variable Create ***SUM.SAS file in ODF directory – ODF summary file necessary for reprocessing Output all SAS environment variables to screen – useful check XMM data reduction: part II

12. Run emproc (if required) Run emchain (if required) Run epproc (if required) Run epchain (if required) List created final calibrated event lists and exit XMM data reduction: part II

13. Run script XMM data reduction: part II

14. Example script : SASprepareXMM XMM data reduction: part II

15. Soft Proton flaring in an observation of a Galaxy Group XMM data reduction: part II

16. Executable t-shell script Comment lines Runs at low priority Number of arguments On-screen help – appears whenever incorrect number of arguments is given (e.g. none) XMM data reduction: part II

17. If correct number of arguments is given (9) then input arguments (1-9) converted to parameters used in script If requested : Check ODF Check for ***SUM.SAS file Create ***SUM.SAS file in ODF Set SAS_ODF environment variable XMM data reduction: part II

18. If requested : Create ccf.cif – calibration index file Set SAS_CCF environment variable Output all SAS environment variables to screen – useful check XMM data reduction: part II

19. Convert patcode parameter into an expression for evselect XMM data reduction: part II

20. MOS1 example (comment) XMM data reduction: part II

21. If $m1 does not equal 0, OK – continue to next line If $m1 equals 0, ignore this whole section XMM data reduction: part II

22. If $m1 equals F, then we attempt to find the file Else, the file name is given by the parameters XMM data reduction: part II

23. Remove a temporary file if it already exists XMM data reduction: part II

24. Long list all the MOS1 event files in the directory to temporary file XMM data reduction: part II

25. Count the number of entries (MOS1 event files) in the temporary file XMM data reduction: part II

26. If only one MOS1 event file, use this XMM data reduction: part II

27. If more than one MOS1 event file, try to find the largest XMM data reduction: part II

28. Initiate loop through ‘i’ (i=0) XMM data reduction: part II

29. loop through ‘i’ until ‘i’ equals number of event files, then end XMM data reduction: part II

30. Increment ‘i’ by +1 (i=1) XMM data reduction: part II

31. Get size of ‘i’th file : Type the file, extract the top (head) ‘i’ lines, then extract the bottom (tail) line – file size is the 5th string in long listing XMM data reduction: part II

32. If size of ‘i’th file is larger than largest recorded, note ‘i’ (‘ibig’) and size as largest recorded XMM data reduction: part II

33. Loop round to next ‘i’ XMM data reduction: part II

34. Get name of largest file (i.e. file corresponding to ‘ibig’) : Type the file, extract the top (head) ‘ibig’ lines, then extract the bottom (tail) line – file name is the 9th string in long listing Set $m1 to this filename XMM data reduction: part II

35. Create (evselect) lightcurve : High-energy, single events, ‘good’ flags, 100s bins XMM data reduction: part II

36. Output help to screen: What is being done, what the user should do XMM data reduction: part II

37. Plot (dsplot) the light curve to the screen for user to examine XMM data reduction: part II

38. Request upper and lower threshold values from user XMM data reduction: part II

39. Use threshold values to create (tabgtigen) good times GTI file XMM data reduction: part II

40. Create (evselect) intermediate event file : filter on low-E threshold, pattern, flags, etc. XMM data reduction: part II

41. Create (evselect) final event file : filter intermediate file for good times using GTI file XMM data reduction: part II

42. Same for MOS2 XMM data reduction: part II

43. Same for pn XMM data reduction: part II

44. XMM data reduction: part II

45. Enter MOS1 thresholds Same for MOS2 Same for pn New filtered event files (r1***EV***) smaller than originals XMM data reduction: part II

46. Already seen xmmselect GUI - clean data, and produce science products Only handles one dataset at a time Essentially a handy GUI ‘wrapper’ to the SAS-task ‘evselect’ evselect is the command-line workhorse of SAS analysis tasks Many scripts are command-line ‘wrappers’ of evselect and other SAS-tasks XMM data reduction: part II

47. Script to create images - Essentially an evselect wrapper, plus other tasks (e.g. asmooth) XMM data reduction: part II

48. e.g. 300-2000eV sky (X/Y) 2′′ images (raw and smoothed) [MOS-sdtq, pn-sd] MOS1 MOS2 pn MOS1 smoothed MOS2 smoothed pn smoothed XMM data reduction: part II

49. Script to create spectra + ARF, RMF and do grouping etc. Again, evselect wrapper, plus arfgen, rmfgen, grppha etc… SAS-task equivalent especget now much improved XMM data reduction: part II

50. createspectrum to create BG spectrum Note: can set parameters (here, co-ordinates from Tim’s region files) on the command line Run script (using parameters) XMM data reduction: part II