1 / 38

Archiving System Status

Archiving System Status. Plan. Archiving Systems Overview HDB / TDB / SNAP / ADB Soleil Deployment Status Oracle DB infrastructure description Latest modifications Production Missing features. Archiving Systems Overview. HDB/TDB Archiving: General Description. Operators Requirements

marny-mccoy
Download Presentation

Archiving System Status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Archiving System Status Archiving Workshop (Soleil, May 2010)

  2. Plan • Archiving Systems Overview • HDB / TDB / SNAP / ADB • Soleil Deployment Status • Oracle DB infrastructure description • Latest modifications • Production • Missing features Archiving Workshop (Soleil, May 2010)

  3. Archiving Systems Overview Archiving Workshop (Soleil, May 2010)

  4. HDB/TDB Archiving: General Description • Operators Requirements • Storage of a large number of data produced by the Control System • Data exploitation at anytime • Watch the vacuum temperature and the pressure evolution • Look for an explanation of a beam loss • The tools • Historical DataBase (HDB) • Data are kept for lifetime • The minimal archiving period is 10 seconds • On-line extraction available during x years (hardware capacity limitation) • Temporary DataBase (TDB) • Data are kept x days (hardware capacity limitation) • The minimal archiving period is 100 milliseconds The two are managed via the graphical tool named Mambo Archiving Workshop (Soleil, May 2010)

  5. HDB/TDB Archiving: Mambo MAMBO : Configuration and Extraction application Archiving Workshop (Soleil, May 2010) Web access also available via JBOSS

  6. HDB/TDB Archiving: Devices • Devices • Hdb/TdbArchiver : Collect of the attributes values and storage in the DB • ArchivingManager : Upper HDB/TDBArchiver, contains Start/Stop archiving commands • HDB/TDBExtractor : various extraction commands as getAttDataBetweenDate, getArchivingMode … • HDB/TDBArchivingWatcher : Supervision Device • Diagnosis tool : detecting abnormal archiving interruption • Recovery : reactivate archiving on the failed attributes Archiving Workshop (Soleil, May 2010)

  7. SNAP Archiving: General Description • Operators Requirements • Capability to take a picture of a set of attributes at t time • Motors positions before a planned electric halt • Compare quickly and easily the attributes values • Before and after an experience to analyse the beamline parameters evolution • Send instructions easily to several equipments • Set the beamline in a configuration reference • The tool • Snap DataBase (SNAP) • Allows to take a snapshot of a set of attributes • Allows to apply values on a set of attributes • Allow to compare attributes values to each other It is managed via the graphical tool named Bensikin Archiving Workshop (Soleil, May 2010)

  8. SNAP Archiving: Bensikin BENSIKIN : Configuration and Exploitation Archiving Workshop (Soleil, May 2010)

  9. SNAP Archiving: Devices • Devices • SnapArchiver : takes the snapshot and stores it in the DB (Hibernate java technology) • SnapManager : triggers the launch snapshot to an available archiver • SnapExtractor : various extraction commands as getSnap Archiving Workshop (Soleil, May 2010)

  10. ADB Archiving: General Description • Operators Requirements • Supervise attribute quality and device state changes • The tool • Alarms DataBase (ADB) • Monitors the attributes alarm state It is managed via the graphical tool named Macarena Archiving Workshop (Soleil, May 2010)

  11. ADB Archiving: Macarena Macarena : Current and Deleted Alarms Archiving Workshop (Soleil, May 2010)

  12. ADB Archiving: Devices • Devices • AdbArchiver : monitors the attributes quality and stores the values in the DB • AdbManager : • dispatches alarm supervision between the existing AdbArchiver devices • Used by the GUI for each database transaction. • AlarmDiary : contains the available alarms list, access by the GUI only Archiving Workshop (Soleil, May 2010)

  13. Common Points • SGAD: GUI tool generates a MySql DB creation script • for Unix or Windows • for HDB, TDB or/and SNAP DB • Common point : • Archiving systems are provided in Java 1.6 Archiving Workshop (Soleil, May 2010)

  14. Soleil Infrastructure Archiving Workshop (Soleil, May 2010)

  15. Machine Deployment status • All the archiving systems use an Oracle Database • SNAP • Oracle engine updated • Following some Bensikin updates and a little ICA pressure the operators plan to work more with it during the next weeks • Currently important data are stored in Matlab files • Still now, the Vacuum group makes some manual comparison at every run startup • ADB • Before the data were stored into files • System recently migrated to be built as HDB/TDB system • GUI consolidation is ongoing and strongly awaiting by the operators to help them in the control system monitoring Archiving Workshop (Soleil, May 2010)

  16. Machine Deployment status • HDB : 8900 attributes • 8750 scalars + 150 spectrums • 1800 at 10 s • 4000 at [30 – 60] s • 1800 at [200 – 900] s • 1000 at [1000 – 3600] s • The rest is upper 3600 s • 25 DServers * 5 devices, so 125 devices hold by One Linux Server • Each device supports about 70/75 attributes (initial estimation was 50 attributes / device) • 5 years of online data • TDB : 6500 attributes • 6470 scalars + 30 spectrums • 10 at 100 ms • 1900 at 500 ms • 3100 at [1 – 5] s • 650 at [10 – 60] • The rest is upper 60 s • 60 DServers * 5 devices, so 300 devices hold by Three Linux Servers • Each device supports about 20/25 attributes (estimation is about 30 attributes / device) • Data are available 30 days Archiving Workshop (Soleil, May 2010)

  17. Beamline Deployment status • All the archiving systems use a MySql Database • SNAP • Really often used to store motor positions, to set beamline in reference position ... • TDB: at most 250 attributes • Mainly scalars • 1 DServers * 5 devices • Data are available 20 days (easily configurable per beamline) • Minimal archiving period equals to 1 s due to a MySql limitation • HDB: Not available • Oracle installation is in progress • All the hardware infrastructure is available • Will be deployed progressively beamline per beamline Archiving Workshop (Soleil, May 2010)

  18. Oracle DB Solution Archiving Workshop (Soleil, May 2010)

  19. Oracle Solution • Refer to the Jérémy’s presentation Archiving Workshop (Soleil, May 2010)

  20. Latest Modifications Archiving Workshop (Soleil, May 2010)

  21. Latest Modifications: Mambo • Mambo • Better .lock file management • At Mambo startup this file is stored in the directory linked to the connexion profile • This file is still available when the application is brutally interrupted  The profile is no more usable • Now .lock can be easily suppressed after confirmation Archiving Workshop (Soleil, May 2010)

  22. Latest Modifications: Mambo • DB connection parameters location • Either in command line with • …-DHDB_HOST=srv1 -DHDB_NAME=hdb -DTDB_HOST=srv1 -DTDB_NAME=tdb fr.soleil.mambo.Mambo … • Useful to start 2 Mambo in front of 2 differents DB • Or in Hdb/Tdb Archiver class properties • DbHost, DbName • Properties also used by the archivers devices Archiving Workshop (Soleil, May 2010)

  23. Latest Modifications: Mambo • Mambo • Display Multiple VC • Some parts has been transformed in java bean : • Table Mode • Connection Window • VC (not completly done) • Start archiving error description in the log part Archiving Workshop (Soleil, May 2010)

  24. Archivage HDB/TDB : Mambo • Archiving modes: something more in the Absolute and Relative modes • Definition • Attribute value is archived when the variation between two successive values (in absolute value or in percentage) is upper configured bounds. • Default behavior • the variation is computed with the attribute value previously read on the device and the current value • Slow drift behavior • the variation is computed with the attribute value previously stored in the DB and the current value Archiving Workshop (Soleil, May 2010)

  25. Latest Modifications: Mambo • Mambo • New Graphical tool usage named Comete  various impacts : • Spectrum viewer • X Axis configuration management Time Stack Indice Time Archiving Workshop (Soleil, May 2010)

  26. Latest Modifications: Mambo • Mambo • Set All Colors in the New/Modify VC Archiving Workshop (Soleil, May 2010)

  27. Latest Modifications: HDB/TDB Devices • HdbArchiver • Some attributes can be archived under 10 s by configuration (not fully tested) • Hdb/TdbArchivingWatcher Device state modification • Number of KO attributes≥ MinNumberOfKoForAlarm (new property) then State = ALARM otherwise State = ON • Database Api refactoring • Done but Not completed Archiving Workshop (Soleil, May 2010)

  28. Latest modifications: Bensikin • At startup: last open snapshot is loaded (quickly opened) • SnapShot description • Contexte number • Number of attributes • Context description • Number of attributes • SetEquipment With Command • Send a command with the Read or Write value as argin Archiving Workshop (Soleil, May 2010)

  29. Latest modifications: Bensikin • SnapShots comparaison • String and boolean attributs types can be also compared • Table bean and Connection window bean usage Archiving Workshop (Soleil, May 2010)

  30. Latest modifications: Bensikin • Application options (proper to the profile) : • Date Filter criteria Snapshot expanded Prefilled Time field Archiving Workshop (Soleil, May 2010)

  31. Latest modifications: Bensikin Application options (proper to the profile) • Columns order definition in the comparison window • Configuration of the Data included in the Snapshot export Archiving Workshop (Soleil, May 2010)

  32. Latest modifications: SNAP Devices • SnapArchiver / Manager • LaunchSnapshot method has been moved to its right place: it concerns the SnapManager and call the TriggerLaunchSnapshot command of the SnapArchiver Archiving Workshop (Soleil, May 2010)

  33. Production Archiving Workshop (Soleil, May 2010)

  34. Maven production <dependency> <groupId>fr.soleil.lib</groupId> <artifactId>hdbtdbArchivingApi</artifactId> </dependency> <dependency> <groupId>fr.soleil.lib</groupId> <artifactId>commonArchivingApi</artifactId> <version>1.4.4</version> </dependency> <dependency> <groupId>fr.esrf.tango</groupId> <artifactId>TangORB</artifactId> </dependency> • Now .classpath project is empty  pom.xml usage • Pom.xml • contains the main jars files •  see Vincent Hardion presentation on Thursday • Archiving packaging ongoing • Either one zip file with the complete archiving systems distribution • Or one zip file per archiving system • Zip file will be stored in the Tango Pink site Archiving Workshop (Soleil, May 2010)

  35. To be improved Archiving Workshop (Soleil, May 2010)

  36. Missing features • No archiving system manage the Tango::Image data_format • Archiving system monitoring must be improved for computer team and operators • Long term TDB: some data mustn’t be deleted • Logging information improvement with log 4 j usage • Database api code refactoring not fully finalized • MySql doesn’t support the milliseconds • Tdb/Mysql DB upgrade (versus Oracle) to ensure a heavy load (eq. SqlLoader, jobs, partitions, backup mechanism …) Archiving Workshop (Soleil, May 2010)

  37. Summary Archiving Workshop (Soleil, May 2010)

  38. To sum-up • Archiving Systems are functionally fully operational • many interesting features are missing • Deployment is a big part of the archiving system activity • we are lacking tools to ease this task • Oracle DB infrastructure management is a huge task • How to share the know-how for MySql management ? • Archiving Systems are always in evolution • Sharing modifications, with the community, is crucial to avoid forking Last but not least : the Archiving system philosophy Give Users the ability to be autonomous Archiving Workshop (Soleil, May 2010)

More Related