260 likes | 288 Views
Detailed instructions on monitoring DCS system, checking data recording, system states, and alarms. Verify histograms, enter data in elog, and take control for corrective actions. Stay vigilant for errors, warnings, and system stability.
E N D
DCS Instructions K. Grogg, M. Weinberg, M. Grothe
DCS Monitoring • Check that monitoring is working • Data is being recorded • Bugs in the code • Check that the system is in a good state • Top level state machine • Histograms for each RMC • Enter information in elog • What was checked and results • Details on any errors, alarms, warnings
Starting • The following slides have step by step instructions for what to look at and what to expect, along with screen shots • Be sure to read Twiki carefully, it has more information than these slides • Start by • Tunnelling • ssh -Y lxplus.cern.ch -L 60001:pcwiscms05.cern.ch:3389 • Set up putty and pvss (See Twiki for full set up instructions!) • This only needs to be done once if you save you settings
Starting PVSS Click here to start
Finite state machine • Are the FSM states of unmasked RMCs either standby or ok? • Any known errors should be masked • RMC10 might be off or in error, usually ok • If need to mask, take control (click lock icon), and change check mark to X • Does the alarm/alert overview panel report any warnings and faults? • Does the alarm/alert history show any new alarm/alert entries? • Need to take control (lock icon) to see • Record any new or fixed alarms • Be sure you have released control when done!
Finite State Machine Take Control Alarm Overview RMC 10 is masked
After taking control Alarm History Release Control Mask/Unmask Do not leave it like this. Be sure to release control!
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 8
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database
Getting to RMC GUI Double click to open RMC GUI
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 11
RMC GUI Panel Check the circled buttons
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 13
Histograms Check all histograms for any changes/problems
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 15
Temperatures Right click here to get this toggle option and select
Temperatures Temperature should be roughly around these values: Temp A ~ 27 ± 3 Temp B ~ 25 ± 3 Temp C ~ 30 ± 3 Temp D ~ 32 ± 3 Axes have been adjusted (dragged) to show all crate temps
Voltage and Analog Temp For most RMCs +5V is usually 5.06 +12V is usually 12.2 -12V is usually -12.3 For RMC 3 +5V is usually 5.7 +12V is usually 14.3 -12V is usually -8.7 Axes have been adjusted to display all voltages Analog Temp should be in the low 20s
Supply Current Current should be between 46-49 A, except for RMC 5 which is usually 50-51 A
Alarm Status Should be zero (0), otherwise there is an alarm!
Online Status Should be one (1) unless there is a known reason otherwise
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 22
RMC GUI Panel Check the circled buttons The values here should match those in the histograms 1/4/2020 23
Closing an RMC GUI Use this button to close, the X doesn’t work
What to expect • Temperatures should be fairly stable • Temp A ~ 27 ± 3 • Temp B ~ 25 ± 3 • Temp C ~ 30 ± 3 • Temp D ~ 32 ± 3 • Look for spikes or slowly rising values • Voltages should be very stable • For most RMCs: • +5V is usually 5.06 • +12V is usually 12.2 • -12V is usually -12.3 • Analog Temp is in low 20s • Supply current should be around 50 • Between 46-49 A, except for RMC 5 which is usually 50-51 A • Alarm Status should be zero • Note any changes and when they occurred • Online/Offline should be one • Note any changes and when they occurred • For RMC 3: • +5V is usually 5.7 • +12V is usually 14.3 • -12V is usually -8.7
In case of problems • Check “Detailed Alarm Status” and “Detailed System Status” • Know where alarms are coming from • Diagnose the problem • Call an expert • In case of Alarm, check “Most Recent Fault Record” to see error info • Check time stamp • If rack power is switched off, some information may not be updated • Do not Refresh All! • Do not click POWER ON, POWER OFF, CLEAR ALARM, or Further Expert Action! • Be sure to enter everything you find in the elog