CSTS-File Transfer service discussions (2)

1. CSTS-File Transfer service discussions (2) CNES position CSTS File transfer service discussions

2. File Content Description • Define manifest files identifying • File content (to the level that the recipient needs to know) • File format? • Processing Instructions • Combine with content description? • Pull Model, Push Model, or both? • [Mouy Francois-Xavier] No needs identified yet for push model (but still a little bit of work to do) • Subscription and automatic push? [Mouy Francois-Xavier] for push model • Subscription and notification of new data? [Mouy Francois-Xavier] how send notifications if provider not bind. Bind return could notify a list of avaliable data • File Management Services (see dedicated slide) • File Directory / Catalogue Servcies • File Management [Mouy Francois-Xavier] for push model • Create / Delete • Move • Local copy • Directory Management [Mouy Francois-Xavier] for push model • Create / Delete / Copy • Storage Control [Mouy Francois-Xavier] needs to be configurable • Duration of Storage • Maximum Volume (per client?) CSTS File transfer service discussions

3. Security (see dedicated slide) • Off the shelf FT protocols support security but must be configured and depending on features used may need supporting infrastructure • Firewall issues • Access Control • At what level? (mission, individual accounts, …) [Mouy Francois-Xavier] Mission level seem to be the right level • Access Control Groups • More complex schemes • What level of privacy / access control is needed? [Mouy Francois-Xavier] We need to be able to provide this kind of features before security person's asks • E.g. can user X see that data are stored for user Y? [Mouy Francois-Xavier] he shouldn't ;-) • Key management issues • Others? CSTS File transfer service discussions

4. Reading Martin initial thoughts. • Protocol choices are wide. If we use one or other protocol, it comes with all this complexity (advantages, drawbacks). • Some features of standards COTS are not useful in our context, but you have to tune and secure those features anyway. • Most of the difficulty come with the push model (security, behavior), the pull model is very near the existing offline model. Initial CNES position: • Firstly, we thought we only need the pull model which is quite simple than the push model. • Furthermore, File transfer function is already done by other ways. (ftp servers in parallel) • Therefore, why redo it ? CSTS File transfer service discussions

5. However, Experience show us that deploying a file transfer feature with common COTS is not quite easy that it is on paper (e.g. with ftp: passive form, active , dynamic port ranges, firewalls, user management, … ). • Using CSTS could simplify the integration because the network security aspects are already done. A part of the reporting too (notifications). • CNES think CSTS could be a good « Secured Pipe » to the Ground center. • A binded instance is maintained (Heartbeat) • The Start/Stop mechanisms could be useful to drive providers behavior • We are able to use CSTS services via a high level of firewalls system CSTS File transfer service discussions

6. Candidate standards (from initial thoughts) • File Transfer Standards • FTP, FTPS, SFTP (SSH based FT) • HTTP / HTTPS ? • Others? • CCSDS Standards • XFDU ? • Others? CSTS File transfer service discussions

7. XFDU (xml formatted data unit) • GOAL : organizing transferred data in an Open Archive environment (OAIS) A container (zip file) with a descripition of its content in a manifest file. The logical description points to the physical content which can be either local or distant CSTS File transfer service discussions

8. XFDU description • The Information Package Map contains the logical view of the package. It’s a hierarchical xml tree representing the content of the package. Each leaf of the tree is a Content Unit and can be referred to from the other parts of the package (i.e. information is linked by the Content Unit reference ID). • The Data Object Section contains all the physical information needed to get the file objects out. • The Metadata Section records all of the metadata for all items in the package. • The Behavior Section associates executable code with the content of the package. CSTS File transfer service discussions

9. XFDU manifest structure The structure map is a hierarchical view of logical references which are pointing to physical data, metadata and specific behaviors. CSTS File transfer service discussions

10. XFDU : benefits • Unique Identification of the data ( ContentUnit ID) • Interoperability by using an XML schema description • Hierarchical structure of the information • Data physical access (DataObject) separated from the logical concepts (ContentUnit) • Metadata associated with the ContentUnit • Specific processes associated with the ContentUnit (BehaviorObject) CSTS File transfer service discussions

11. XFDU : drawbacks XFDU developed for approaching needs but not totally the sames (multi-parts, files, cdrom) • Few implementations due to standard complexity.: • NASA XFDU library, ESA SAFE’s archive for ENVISAT data, CNES’s PAIMAS • Behavior function not really validated CSTS File transfer service discussions

12. CSTS-XFDU : conclusion • The data organization has been studied earlier by the MOIMS/IPR group. XFDU is an answer to csts concerns with the following conditions: • Needs of an XFDU « light » to answer the requirements. • Needs to study the behavior statement. . CSTS File transfer service discussions

13. CSTS-XFDU : conclusion • XFDU (simplified one) Could be a good system to integrate the file transfer feature. • The behavior side of XFDU could be a response to File transfer and more … • CNES thinks it could be a possible cooperation and some people are interested to study those aspects … CSTS File transfer service discussions

14. CSTS File transfer service discussions

15. Some use cases at CNES • Distant management of stations • Management by xfdu of TM archived in station • TM playback via CSTS-offline service • Inter-facilities scheduler • Distant scripts used by the scheduler managed by xfdu • Launch via services CSTS • Re-use of CNES well-known scheduler GUI • … CSTS File transfer service discussions

16. Let’s take an example…. XFDU over CSTS for playing recorded telemetry CSTS File transfer service discussions

17. Telemetry transfert and play • A facility wants to transfert simulated telemetry to a station and to be able to play it back later. • The user build an xfdu package (X1) with the TM file and two scripts (play and rewind) • All the configuration and security concerns to be solved by the management layer. CSTS File transfer service discussions

18. STEP 1CONNEXION user provider CSTS FT CSTS FT bind XFDU X1 TM 1 PLAY REW CSTS File transfer service discussions

19. STEP 2XFDU TRANSFERT user provider CSTS FT CSTS FT XFDU X1 bind TM 1 START PUT X1 XFDU X1 TM 1 PLAY REW PLAY REW CSTS File transfer service discussions

20. STEP 3XFDU AUTOMATICINSTALL user provider CSTS FT CSTS FT XFDU X1 bind TM 1 START PUT X1 XFDU X1 TM 1 PLAY INSTALL REW PLAY REW PLAY REW TM 1 CSTS File transfer service discussions

21. STEP 4The user lists all available xfdusand their behaviors user provider CSTS FT CSTS FT XFDU X1 bind TM 1 START PUT X1 XFDU X1 TM 1 PLAY INSTALL START LIST REW X1 PLAY START LIST X1 REW PLAY PLAY, REW REW TM 1 CSTS File transfer service discussions

22. STEP 5The user plays the TM1 telemetry back user provider CSTS FT CSTS FT XFDU X1 bind TM 1 START PUT X1 XFDU X1 TM 1 PLAY INSTALL START LIST REW X1 PLAY START LIST X1 REW PLAY PLAY, REW REW START RUN X1 PLAY TM 1 TM 1 CSTS File transfer service discussions