470 likes | 538 Views
Planck SGS Implementation Review January 24 nd -25 th January 2007 @ ESTEC. HFI SGS Overall Status. HFI context Development philosophy Major events since SGS DR DR recommendation status Organisation/Management SGS1 Overview SGS2 Overview Hardware Infrastructures
E N D
Planck SGS Implementation Review January 24nd-25th January 2007 @ ESTEC HFI SGS Overall Status HFI context Development philosophy Major events since SGS DR DR recommendation status Organisation/Management SGS1 Overview SGS2 Overview Hardware Infrastructures Software Infrastructures Future Milestones Resources & Schedule DPC Goals & Scientific exploitation Conclusions François R. Bouchet - HFI Deputy PI & DPCM
PLANCK vs WMAP:1 year Sensitivity Goals The aggregated sensitivity of 3 of the core CMB channels of Planck, @ 100, 143, 217GHz (~0.5μK.deg in T, 1 μK.deg QU) will be unprecedented and quite challenging in terms of control of systematics! F. R. Bouchet - HFI SGS Overall Status
HFI Context • Calibration has shown that HFI should reach its very ambitious sensitivity goals • Cooling chain should allow 4 sky surveys (if tanks are fully filled). This would provide • Increased redundancy (systematics) • Improved sensitivity • Both could mean the difference between B mode detection or not • Emphasis for extracting some key instrumental parameters has somewhat shifted towards sky data analysis (beam, polarisation). F. R. Bouchet - HFI SGS Overall Status
Development philosophy • For the scientific analysis part, have a toolbox (modules & sub-pipes) with pieces demonstrated to work together (pipeline release), implementing basic needed functions and full infrastructures, all tested on simulations • Plan for, in flight, a progressive approach for the processing, based on a succession of releases, each building on accrued knowledge in previous iteration (even without extra data) IMO; need for flexible environment, early C(l) as quality assessment tool Details in “Planck HFI Data Processing”, ref. PL-HFI-IAP-PL-000 Date: 2004-09-17 F. R. Bouchet - HFI SGS Overall Status
Major Events since DR in Feb. 2005 • SGS-DRRecommendations mostly Implemented (cf. full assessment in Annex) • A new organisation & a more formalised way of working introduced & followed after DR • HFI Core Team created in January 2006 • A new GS Project manager hired following end of contract of L. Sauter at end of August; Organisation change put in place in Nov06 > obvious difficulty • Extensive use & real life qualification tests of SGS1 elements (L1, DMC1, QLA…) thanks to CQM + PFM testing & Calibration • > (6m, 50 000TC, 500GB), including 19 days of science data @100mK at full throughput; scientific results presented @ ISVR ~3 months later. • Fully qualified DM release at end of 2006 (finalised test report recently released) • HFI DMC-V2 designed/developed/tested/IDIS integrated… • IDIS V2.3 (inc HFI-DMC V2.0) acceptance tests started on Jan 15th Based on ESA planning,Nov06 F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendations Status F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendation Status 1/3 • Complete status in Annex. Here only the not closed ones which have not been already covered by PSO. • S20 - Develop thorough documentation of all aspects of the project and implement succession planning for those junior scientists who will take the place of their seniors. Ongoing - Started recently. Living Calibration document following ISVR. Idem for IMO document • P3.1 (critical/ medium term) The DPCs shall define instrument data quality indicators [by May 2005 in view of the official launch slip to August 2007. However SPPT development needs some indication of the nature and the format of these indicators—ICD information—a bit earlier, say Feb 2005]. Ongoing - DQR ICD released, WQR definition being finalized (production will occur after IOT/L2 weekly meetings) • P5.2 (critical/ long term) The PSO and the DPCs shall establish and agree upon the process, timeline and required resources to facilitate the capture, transfer, and retention of the appropriate knowledge about the mission and the resulting data, in order to allow for effective support to users of the Planck Archive in the years following the mission. We agreed only to a WMAP-like scheme. No extra funding requested. – (In survey mission, Data Release & support evolved) (but planning 2 releases). There might be a need to fill for specialist support, by specialists. F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendation Status 2/3 • P6.1 (critical/ urgent) The Planck Project and the Planck Ground Segment System-engineering Group (PGSSG) shall urgently review and agree the scope and schedule of the first interface tests, currently planned in mid-2005. See also RID GSDR-050 from MOC Design Review Ongoing - Scope & schedule discussed within PGSSD. Cf. next talk. • P8.1 (routine/ medium term) The HFI DPC shall make a detailed plan with timelines for procurement, installation and deployment of the necessary hardware and software to run their pipeline in due time. Ongoing - Cf. HSADD for dimensioning (stable). Will reassess again detailed plans after 6 months of M2 operation.& PreFM deployment • O5.1routine/ long term The instrument teams shall produce the chapters of the Instrument Users Manual concerned with the Flight Operations Procedures as per annex 1.5 of the OIRD. Ongoing - Elementary procedures written. Need to be described/assembled in the User manual. F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendation Status 3/3 • O6.1 (routine/ medium term) The severity of potential problems arising from the usage of different S2K versions throughout the Ground Segment shall be assessed, either in the recently created HPSBD task force, or in the Science Engineering Working Group(s). See also RID GSDR-004 from MOC Design Review. Identified by HFI as an important SGS risk. Solution not agreed yet. • O7.1 (routine/ medium term) The DPCs shall generate scenarios for the operation of the pipelines at the various levels that describe the amount of automation versus the need for human intervention during routine processing, and update their resource estimates accordingly; these scenarios should be based on actual test runs with typical data volumes that realistically simulate several days or a few weeks of pipeline operations and related interface activities. Ongoing - Continuing process. Detailed assessments done during Calibration and with DM. FM next. Not viewed as outstanding problem. F. R. Bouchet - HFI SGS Overall Status
New HFI organisation • HFI Core Team= HFI forum for debates involving performances and operations, systematic effects, data processing algorithms to produce deliverables • It gathers all HFI instrumentalists, data analysts and theorists collaborating in order to perform the analyses leading in the end to a successfully delivery of scientifically usable HFI data to the HFI consortium, to the Planck collaboration, and later to the entire scientific community. • It is organised in ad hoc subgroups for the tasks at hand, people can be in and out for a while. • Core Team Meetings(6 a year so far) • CT 01 / Jan 2006 @ IAS / Kick off meeting • CT 02 / Mar 2006 @ IAS / Calibration preparations • CT 03 / Jun 2006 @ IAS / Calibration Plan finalization • CT 04 / Jun 2006 @ IAP / DPC Development Model status and plans • CT 05 / Sep 2006 @ IAS / Calibration current results • CT 06 / Dec 2006 @ IAP / DPC: SGS review preparation & overall status • CT 07 / Feb 2007 @ IAS + CT 08 / Apr 2007 @ IAP + CT09 / June 2007 + … • In flight, we plan for • monthly meetings, with all groups reporting on their activity; • common LFI-HFI Core Teams at 6 (TBC) months interval. • This is when analyses will be reported and discussed, actions decided by the CT management group, in order to update the version of the IMO (Cf. Desert’s talk) & of HFI pipeline segments. This sets a natural time scale for internal releases & updates. • It also defines a management scheme for IMO updates & operations timeline, report to PSO. • Planck “Transverse Working groups” (1=SEWG, 2=Comp-Sep, 3= CTP) are common between HFI & LFI Core teams and serve as R&D groups for the DPC F. R. Bouchet - HFI SGS Overall Status
HFI scientific organisation Post instrument delivery & SGS Manager nomination HFI members of the Planck Science Team HFI Steering Group R. Bond, P. de Bernardis, G. Efstathiou, K. Bennett, J. M. Lamarre, A. Lange, S. White, P. Ade PI J.-L. Puget Deputy PI F.R. Bouchet Planck Working Groups 4-8 Planck Working Groups 1-3 HFI Core Team Instrument Working Group M. Piat & P. Ade & A. Lange & B. Maffeï & J. Bock & D. Yvon Instrument scientist J.-M. Lamarre Survey scientist G. Efstathiou HFI Core Team Management Group Puget + Lamarre & Efstathiou & Lange & Bouchet & Ashdown & Désert & Ganga & Guyot & Vibert DPC Scientific Coordination Group M Ashdown & FX Désert & K Ganga DPC manager F. Bouchet Instrument project mgr G. Guyot SGS project manager L. Vibert November 13th 2006 see Instrument organization see SGS organization See next talk on management & SGS1 by Vibert/Puget F. R. Bouchet - HFI SGS Overall Status
Risks management • We deployed after the DR a detailed tracking of risks using a web tool where all GS actors could enter risks; assessment and follow up of the identified risks was not satisfactory for lack of dedicated risk managers • After 1.5 years of use, we found this system to be inadequate as is, and propose instead an identification of major risks by the management(following the model used by the instrument), to be regularly updated and reported to ESA, as discussed at PST yesterday. • All posted risks were reviewed. We consider we can close most of then (either due to progress, or unaccepted by new SGS manager). • Only one remaining major risk in the HFI-SGS at present (ProC next step). • NB: we also identified another risk, at the Planck SGS level, pertaining to the different SCOS versions used. F. R. Bouchet - HFI SGS Overall Status
SGS2 Overview • Objectives for the main releases • DM: get a first complete pipeline to insure no show stopper, on quasi-ideal data (little on polarisation, MC for errors) • FM: add polarisation and full Monte-Carlo abilities, start coping with obvious non-idealities (time evolution of gain/noise, of star tracker to FP geometry) • Getting there: • Detailed implementation & test plans (in progress for FM) • Regular builds (~weekly) on magique-2 and CCALI • An internal release every 6 months (e.g. preDM) • A strong R&D program in CT, notably WG1-3 • DM released Dec06, tested & fulfilling objectives (butDM12), FM in Dec07 (preFM in April on basis of DM+IDIS v2.3). • “End-to-end” testing plan developed (cf. PSO). Practical implementations being worked out. Polkapix F. R. Bouchet - HFI SGS Overall Status
DM Pipeline design • Includes: • Central IMO role • Pre-processing, starting from L1 • Beams & FPG reconstruction • (write quaternions to IMO) • Noise characterisation on TOIs • Map-making (destriper) & flux • calibration on orbital dipole • Component separation & • generation of final catalogue • Power spectrum estimation • (with simple MC loops) IMO F. R. Bouchet - HFI SGS Overall Status
WG2, WG3 & DPC Codes CTP (WG3): map-making codes comparison, cf. talk by O. Perdereau NB: power spectrum code (ongoing) comparison cf. talk by E. Hivon WG2: Blind Component Separation Challenge, as of Jan 07 cf. talk by M. Ashdown Current HFI DPC code WG2&3 thus provides useful guidance / performance verification to HFI DPC F. R. Bouchet - HFI SGS Overall Status
HFI SGS Hardware • Creation of the Planck HFI operation room was part of the remodelling effort of the IDOC at IAS centre, in the fall of 2006. The room is now available, and furniture and computers are being installed at present. • New computer room @ IAP, dedicated to Planck HFI DPC was delivered in time, at the end of November 2006. Everything is OK after reception & minor adjustments. • Magique-2 extension (adding 128 dual core processors with 512 GB RAM, 1030TB disks) was delivered one month late (end of December instead of end of November). One non-conformance is on Aggregated Write performance. It will be operational to all DPC users end of January, for starting DPC (pre-) Flight Model development. • 3rd stage of plan: M3 planned deployment during cruise to L2 (Cf. HSADD) • CCALI (large PC farm + Storage Silo) now routinely used for backup and generating simulations since early 2005; practicing transfers of Tera-Bytes of data every other month. • Additional hardware available/planned @CPAC,USPDC & for R&D (e.g. NERSC/WG3) Operation room @ IDOC Magique-2 fully deployed Space ready for Magique-3 F. R. Bouchet - HFI SGS Overall Status
Software infrastructures 1/2 • Web tools (available since Level2 pre-DM) • interface to manage CVS access rights (for concurrent code development) • interface to announce new package tags, and store them in a “mysql” database • automatic presentation of summary of release built & logs • since pre-DM -> debugging of all this (i.e. quite stable by now) • CMT tools (CMT = Configuration management Tool), on 5 platforms (M2 & CCALI since pre-DM + USPDC & MPA since Nov06 + generic Linux, still in development) • CMT “glue” packages describing external libraries on 3 platforms (M2, CCALI, USPDC, with small subset for IDIS at MPA) • development of release build scripts on all platforms • v1 available & running since pre-DM • v2 in beta test now, will be used for pre-FM) • possibility to automate release build from new announces (not used at the moment) • Interface layer around ProC to integrate it into the HFI environment (including ProC patches to be able to use MPI and PBS) • ingestion of release content (including external libs) in DMC + WEB interface: in development • General utility libraries • PIE generates the module code for reading parameters with HFI-DMC, from their ProC interface description • version compatible for HFI-DMC v1 (aka piolib) since pre-DM • version compatible for HFI-DMC v2 being debugged. • Pointing Lib, available since pre-DM • implemented in some L2 module in DM implementation • in all L2 modules in pre-FM • Support team (established beginning of 2005, for pre-DM). • Mailing list server (available since 2004) Close to maintenance mode for all these F. R. Bouchet - HFI SGS Overall Status
Software infrastructures 2/2 • LS: introduced a CCB to prioritize development → OK • L4: initially only final ingestion in FINDAS. Extension to simple distribution tool. URD being developed → OK for this stage (see talk by T. Ensslin) • IDIS common tools (see talk by J. Sternberg on IDIS) • Documents management, in maintenance (Livelink) • Code management, in maintenance (CVS) • Federation layer (authorisation + identification – useful, but not critical) OK for HFI as is (implies a minor internal DMC change, planned) • ProC, is tightly connected to the DMC, see below • HFI DMC is developed by HFI, following the specific requirements imposed by data volume and speed of I/O, status follows • HFI DMC since last SGS review (+ 2 persons Paid by CNES since DR): • V1 (PIOLIB) has been extensively used to acquire HFI data during all tests, and to process data in the DPC pipelines (including DM) • V2 (actually v02-02-46) has been /designed/developed/tested (cf. on-line test report http://cvs.planck.fr/cvs/Level2/Lib_pkg/HL2_DMC/cmt/magique2/PLResult.html?revision=1.74&pathrev=v02-02-46 ), and integrated with IDIS/ProC for V2.3; V2.3 is in acceptance testing since Jan 15th 2007. • IDIS/ProC • Its development is trying to achieve more than needed for HFI alone (and probably LFI) • An important step is lying ahead, foremost loops, but also scheduler at all HFI centres, streaming… with a very constrained schedule (pre-FM in Apr, FM1 in Dec07) • This present an important element of risk, since it can stymie the work of all dependent users. • It has been identified by HFI as its major SGS risk element today, which must be actively controlled. F. R. Bouchet - HFI SGS Overall Status
Future Milestones • Details of SGS2 End-to-end testing is being worked out under the auspices of PSO; goals is to have phase 2 finished for the end of 2007 F. R. Bouchet - HFI SGS Overall Status
Resources Status & Schedule • Resources: • CNES: OK • PPARC: OK for Ops-post Ops; 2-3 FTEs for integration and testing from now to launch need to be formally acknowledged • NASA: OK • DLR: ? (internal reallocation?) for ProC finalization with all required critical functions in time • CSA: OK • RSSD: OK • Need formal confirmation of funding of Ops/post-Ops phase by the concerned agencies • Overall schedule: • We could operate HFI with all SGS1 aspects by the end of 07 • by the end of 2007, SGS2 must go into a phase dominated by algorithmic development and end to end testing until launch • After launch, shift to flight operations, with iterations (typically 6 months apart) of pipeline major versions and data release (see next slide) F. R. Bouchet - HFI SGS Overall Status
DPC Goals & Scientific Exploitation • Goals (TBC) for DPC Data deliveries: • It is very tight. A priori release criteria not easy to define • Plan for the exploitation: • people in WG4 to WG8 will work with well defined data products, similar to WMAP ones (WG1-3 will merge into the Core Teams), • and those needing in-depth expertise for their science will need to include a knowledgeable member of the CT to help. Not adding an extra burden on the DPCs will actually be an important criteria in picking up teams for different scientific topics. • For the public release(s), also similar to WMAP one. F. R. Bouchet - HFI SGS Overall Status
Conclusions • Our assessment is that the situation is globally OK, given the objectives of the review, although some reorganisation is still ongoing at review time. • Will provide more detailed conclusions at the end of HFI presentations F. R. Bouchet - HFI SGS Overall Status
HFI Conclusions • During the calibration campaign, HFI was operated, monitored and its data stored at full throughput; that data was analysed for the ISVR within less than 4 months. • Operation interfaces defined, many in place, none being problematic • Basic SGS2 infrastructures and pipelines (DM) are falling into place • Strong R&D in CT & WG1-3, with good DPC coupling • Need to switch the efforts ASAP from SGS1 to further algorithmic developments controlling the ultimate accuracy attainable, with corresponding tracking • HFI Core team is in place and working, covering the large span of expertise needed to secure at best HFI SGS to deliver to the Planck collaboration and then to ESA the best possible scientific product… • But we had a lack of management over the last 6 months; the reorganisation has now been completed and a new project management has been defined, but we did not yet fully catch up by the time of this review. We aim to accomplish that in less than 3 months. F. R. Bouchet - HFI SGS Overall Status
ANNEX F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendations Status 1/6 Colour coding: • Transparent = NA (Non Applicable to HFI specifically) • Green = “Closed”, • Orange = “ongoing” • Red = problematic… ID key: B = Board S = Science Panel P = Programmatic from Technical Panel M = Methodology from Technical Panel O = Operations from Technical Panel F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendations Status 2/6 F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendations Status 3/6 F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendations Status 4/6 F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendations Status 5/6 F. R. Bouchet - HFI SGS Overall Status
SGS-DR Recommendations Status 6/6 F. R. Bouchet - HFI SGS Overall Status
ACRONYMS LIST F. R. Bouchet - HFI SGS Overall Status
HFI assessment of main RISK • The IDIS/ProC development is trying to achieve more than needed for HFI alone. E.g.: • Another DMC, AND an imposed choice to define common interfaces (LLIO) rather than ifdef (recall than requirements are different, since HFI has ~10 times more data than LFI, and many processing steps include reading timelines. Also better sensitivity may require additional or more demanding steps) • Desire to allow running on the GRID and create a reusable product • Desire to have a centralised identification/authorisation system (FL). • In brief, a desire to be generic with substantial overhead in some places (Paris), resulting in changes which are complex/slow to implement. • An important step is lying ahead, foremost loops, but also scheduler at HFI centres, streaming… • requiring tight interactions, • a short time available (People need to implement Monte-Carlo in DPC environment to be flight ready) • As a result, HFI believes that the new step in the year ahead (till FR review) calls for an improved way of working. Proposal: • Implement much tighter technical interactions, essentially every week, for 2 months, and assess the situation then • Develop in parallel a less ambitious, but realistic B plan, focusing on the essential for HFI, not to loose too much time in detailed design is 1) fails F. R. Bouchet - HFI SGS Overall Status
Statistics • Downtime of magique 2 in 2006: • Days concerned = 10% (36 days, but interruptions do not compulsory last 24hours) • Unplanned Days concerned = 5% (16 days) • PBS Usage of Magique2 in 2006: • MPA all: 64/14399 = 0,4% of all jobs • Delouis: 2452/14399 = 17% of all jobs • SPR on mantis from June 2006 to Jan 15 2007 (6 months) • 61 SPR on ProC; 32 are closed, 23 are resolved, 6 are open • HFI posted: 43/61 • LFI posted: 0/61 • ESTEC/MPA: 18/61 • NB: the Proc/DMC interface at the origin of 5/61 • SCR on mantis from June 2006 to Jan 15 2007 (6 months) • 20 SCR on ProC; 0 closed, 7 resolved (accepted), 13 open • HFI posted: 19/20 • MPA posted (mreinecke) 1/20 • NB: the Proc/DMC interface at the origin of 1/20 F. R. Bouchet - HFI SGS Overall Status
HFI Instrument organisation F. R. Bouchet - HFI SGS Overall Status
HFI SGS Organisational chart F. R. Bouchet - HFI SGS Overall Status
a) To verify that the development and implementation status of SGS1 is commensurate with the project's scientific, technical and operational requirements. To achieve this the Board will: establish whether the design of all SGS1 elements has been frozen and is compatible with all requirements and interfaces. assess the status of SGS1 interfaces, both internal to the various operational centers of the SGS and external with the MOC, with respect to completeness and consistency. assess the preparation of operations plans and verify that they are consistent with the design of the system. assess the integration/verification/validation plans and their state of implementation, and verify that they are compatible with the overall schedule. verify that the infrastructure supporting SGS1 is adequate. assess PA/QA aspects, verifying that the necessary mechanisms are in place and being followed. assess the ability of the SGS1 system to meet the needs of the Project (completeness, technical and operational performances), in light of the available test results. b) To demonstrate compatibility with managerial aspects, specially in terms of required resources and schedule. In addition, the review will ensure that all the actions arising from the "Planck SGS DesignReview" (held in December 2004) applicable to SGS1, have been closed and all the recommendations properly addressed. a) To verify that the design and development status of SGS2 is commensurate with the project's scientific objectives, is compatible with all requirements, and feasible within all constraints. To achieve this the Board will: • assess the current design and implementation of the scientific pipelines, and their planned evolution, against the full set of objectives of the mission • assess operational aspects and verify that they have been planned for and included in the design of the system • ensure that verification/validation plans have been established and are being followed; assess available results • assess performance (technical and scientific, current and future) on the basis of analysis, and tests on prototypes b) To demonstrate compatibility with managerial aspects, especially in terms of available resources and required schedule (i.e. evaluate schedule against system-level needs; manpower deployment plans; hardware procurement) c) To assess the overall ability of SGS2 to deliver to ESA the formal scientific products of the mission d) To assess the overall ability of SGS2 to enable the scientific exploitation of the Planck data (both pre- and post- product delivery) In addition, the review will ensure that all the actions arising from the "Planck SGS Requirements Review" (held in December 2002) and applicable to SGS2, have been closed and all the recommendations properly addressed. Review Objectives SGS1/2 F. R. Bouchet - HFI SGS Overall Status
Documentation/Organisation • Since DR, we have followed a much more structured documentation approach. • One drawback of the implementation has been that information about SGS activities has been spread among many documents. For obvious reasons these documents were then not updated at the same time and with the same level of granularity and quality thus leading to some incoherence and out-of-date descriptions. • By re-assembling information in a single document, the aim is to ensure again a global coherence. • It is proposed to gather all information related to activities and planning in a document called Planck HFI SGS Work Organization Plan [PL-HFI-IAS-PL-WOP], available on Livelink. • Moreover, because work activities are dynamic by nature, the document will be a living document. The target is to update is as often as needed. • At the Quality process level, the WOP will then supersede the following documents thus making them obsolete: • Planck HFI Schedule and Development Plan • Planck HFI DPC Work Breakdown Structure • Planck HFI DPC Deliverable Document List • Planck_SGS1_CI_List_v4.0_20_11_2006 F. R. Bouchet - HFI SGS Overall Status
SGS1 Overview • Review and update of the operation plan, tasks and tools done, in view of our understanding of the performances of the FM instrument • Overall status of the main tools: • L1 to be fully tested by June, acceptance on flight operations machines in September • transfer responsibility as planned from APC (calib. v.) to IAS (flight v.) • write the flight version by removing calibration specific modules, completing flight specific modules, • rewriting some of the modules (libraries) which generate minor problems (identified during calibration and system test) and weakens the very high stability required for L1 • SGS interface specific s/w tools (not part of L1): mostly files exchange through HPFTS, ICDs stable, system aspects inside HFI-SGS to be consolidated (circulation and processing and decision process around these information) • Scientific operation tools (RTA, QLA, TA) extensively used in calibration and system tests. Almost completed for flight critical elements. Very satisfactory situation. We propose a light management structure at system level (HFI internal, light reporting to project, remove from configuration control at H-P level. • Next talk by Vibert/Puget devoted to SGS1 IOT/DPC Status (in addition to overall SGS management) F. R. Bouchet - HFI SGS Overall Status