HMI-AIA Joint Science Operations Center Science Data Processing a.k.a. JSOC-SDP

1. HMI-AIA Joint Science Operations Center Science Data Processing a.k.a. JSOC-SDP Overview

2. House- keeping Database Primary Archive Catalog Offsite Archive LMSAL HMI & AIA JSOC Concept GSFC White Sands LMSAL MOC SDOGS DDS HMI & AIA Operations Stanford HMI JSOC Pipeline Processing System Redundant Data Capture System Quicklook Viewing 10-Day Archive Local Archive AIA Analysis System High-Level Data Import Offline Archive Data Export & Web Service World Science Team Forecast Centers EPO Public

3. JSOC Dataflow Rates

4. JSOC-SDP Major Components

5. JSOC DRMS/SUMS Basic Concepts • Each “image” is stored as a record in a data “series”. • There will be many series: e.g. hmi_ground.lev0 is ground test data • The image metadata is stored in a relational database – our Data Record Management System (DRMS) • The image data is stored in SUMS (Storage Unit Management System) which itself has database tables to manage its millions of files. • SUMS owns the disk and tape resources. • Users interact with DRMS via a programming lauguage, e.g. c, FORTRAN, IDL. • The “name” of a dataset is actually a query in a simplified DRMS naming language that also allows general SQL clauses. • Efficient use of the system relies on direct use of DRMS • Data may be exported from DRMS as FITS or other (TBD) protocols for remote users. • Several Remote DRMS (RDRMS?) sites will be established which will “subscribe” to series of their choice. They will maintain RSUMS containing their local series and cached JSOC series. • The JSOC may act as an RDRMS to access products made at remote sites.

6. Pipeline Program, “module” JSOC Science Libraries Utility Libraries DRMS Library Record Cache Record Management Keyword Access Link Management Data Access JSOC Pipeline Processing System Components Pipeline processing plan Pipeline Operator SUMS Disks Processing script, “mapfile” List of pipeline modules with needed datasets for input, output PUI Pipeline User Interface SUMS Storage Unit Management System DRMS Data Record Management System SUMS Tape Farm Processing History Log Database Server

7. Simple example – find and look at an image Example of simple utility “module” called “show_”keys” First find images in the minute starting 9 Sept at 23:50 then look at one with ds9 P% P% show_keys "ds=hmi_ground.lev0[][2007.09.09_23:50/1m]" key=FSN,T_OBS FSN T_OBS 0566684 2007.09.09_23:50:01_UTC 0566685 2007.09.09_23:50:06_UTC 0566686 2007.09.09_23:50:11_UTC 0566687 2007.09.09_23:50:16_UTC 0566688 2007.09.09_23:50:21_UTC 0566689 2007.09.09_23:50:26_UTC 0566690 2007.09.09_23:50:31_UTC 0566691 2007.09.09_23:50:36_UTC 0566692 2007.09.09_23:50:41_UTC 0566693 2007.09.09_23:50:46_UTC 0566694 2007.09.09_23:50:51_UTC 0566695 2007.09.09_23:50:56_UTC P% P% ds9 `show_keys "ds=hmi_ground.lev0[566686]" seg=file -p -q` P%

8. JSOC Export • JSOC will support VSO access • JSOC will also have a direct web access • There will be remote DRMS/SUMS systems at key Co-I institutions • ALL HMI and AIA data will be available for export at level-0 through standard products (level-1 for both and level-2 for HMI) • It would be unwise to expect to export all of the data. It is simply not a reasonable thing to expect and would be a waste or resources. • Our goal is to make all useful data easily accessible. • This means “we” must develop browse and search tools to help generate efficient data export requests.

9. DRMS/SUMS Configuration • DRMS and SUMS use the open-source PostgreSQL database engine. • DRMS will run on a pair of dedicated servers likely 4-quad-core processors with up to 10TB fast disk. • SUMS will consist of file servers with attached tape systems. • SUMS will manage 200TB cache disk with 150TB/year of permanent archive for level-1 and higher level products. • SUMS archive will use LTO-4 tapes in a robotic system with at least 10 drives and 2000 tapes near-line. • The pipeline processing system will have about 50 processor cores dedicated to level-0 to level-1 processing and about 450 cores for higher level processing in the pipeline. HMI standard products will need about half of these.

10. Extra Info

11. Pipeline client process Analysis code C/Fortran/IDL/Matlab OpenRecords CloseRecords GetKeyword, SetKeyword GetLink, SetLink OpenDataSegment CloseDataSegment JSOC Library Record Cache (Keywords+Links+Data paths) Pipeline client-server architecture Generic file I/O Data Segment I/O JSOC Disks JSOC Disks JSOC Disks JSOC Disks DRMS socket protocol Data Record Management Service (DRMS) Data Record Management Service (DRMS) Storage unit transfer Storage Unit Management Service (SUMS) Data Record Management Service (DRMS) AllocUnit GetUnit PutUnit Storage unit transfer SQL queries Tape Archive Service PostgreSQL Database Server SQL queries SQL queries Record Catalogs Record Catalogs Series Tables Record Tables Storage Unit Tables

12. Pipeline batch processing • A pipeline batch is encapsulated in a single database transaction, “DRMS session”: • If no module fails all data records are commited and become visible to other clients of the JSOC catalog at the end of the session • If failure occurs all data records are deleted and the database rolled back • It is possible to commit data produced up to intermediate checkpoints during sessions Pipeline batch = atomic transaction Module 2.1 Module N Commit Data & Deregister Module 1 Register session … DRMS API DRMS API DRMS API DRMS API DRMS API Module 2.2 DRMS API Input data records Output data records DRMS Service = Session Master Record & Series Database SUMS

13. Codes to be developed at Stanford Codes being developed in the community Codes to be developed at HAO Standalone production codes in use at Stanford MDI pipeline modules exist HMI module status and MDI heritage Intermediate and high level data products Primary observables Internal rotation Heliographic Doppler velocity maps Spherical Harmonic Time series Mode frequencies And splitting Internal sound speed Full-disk velocity, sound speed, Maps (0-30Mm) Local wave frequency shifts Ring diagrams Doppler Velocity Carrington synoptic v and cs maps (0-30Mm) Time-distance Cross-covariance function Tracked Tiles Of Dopplergrams Wave travel times High-resolution v and cs maps (0-30Mm) Research codes in use by team Egression and Ingression maps Wave phase shift maps Deep-focus v and cs maps (0-200Mm) Far-side activity index Stokes I,V Line-of-sight Magnetograms Line-of-Sight Magnetic Field Maps Stokes I,Q,U,V Full-disk 10-min Averaged maps Vector Magnetograms Fast algorithm Vector Magnetic Field Maps Vector Magnetograms Inversion algorithm Coronal magnetic Field Extrapolations Tracked Tiles Tracked full-disk 1-hour averaged Continuum maps Coronal and Solar wind models Continuum Brightness Solar limb parameters Brightness feature maps Brightness Images

14. AIA Level-2

15. JSOC Data Volumes from Proposal

16. Sample of DRMS/SUMS Resource Assessment

17. JSOC Level-0 processing flow

18. JSOC-SDP Status • Capture system complete, waiting testing with DDS at White Sands • DRMS and SUMS virtually done • Level-0 work progressing, to be done by instrument deliveries • Level-1 work to start after delivery • Getting serious about work on basic pipeline modules • Review of status and computer hardware plans in early November

19. Stanford JSOC effort plan