1 / 53

Scaling Up PVSS

Scaling Up PVSS. Showstopper Tests Paul Burkimsher IT-CO. Aim of the Scaling Up Project WYSIWYAF. Investigate functionality and performance of large PVSS systems Reassure ourselves that PVSS scales to support large systems Provide detail rather than bland reassurances.

sadah
Download Presentation

Scaling Up PVSS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Scaling Up PVSS Showstopper Tests Paul Burkimsher IT-CO P.C. Burkimsher IT-CO-BE July 2004

  2. Aim of the Scaling Up ProjectWYSIWYAF • Investigate functionality and performance of large PVSS systems • Reassure ourselves that PVSS scales to support large systems • Provide detail rather than bland reassurances

  3. What has been achieved? • 18 months PVSS gone through many pre-release versions • “2.13” • 3.0Alpha • 3.0Pre-Beta • 3.0Beta • 3.0RC1 • 3.0RC1.5 • Lots of feedback to ETM. • ETM have incorporated • Design fixes & Bug fixes

  4. Progress of the project • Has closely followed the different versions. Some going over the same ground, repeating tests as bugs were fixed. • Good news: V3.0 Official Release is now here (even 3.0.1) • Aim of this talk: • Summarise where we’ve got to today. • Show that the list of potential “showstoppers” has been addressed

  5. What were the potential showstoppers? • Basic functionality • Synchronised types in V2 ! • Sheer number of systems • Can the implementation cope? • Sheer number of displays • Alert Avalanches • How does PVSS degrade? • Is load of many Alerts reasonable? • Is load of many Trends reasonable?

  6. What were the potential showstoppers? • Basic functionality • Synchronised types in V2! • Sheer number of systems • Can the implementation cope? • Alert Avalanches • How does PVSS degrade? • Is load of many Alerts reasonable? • Is load of many Trends reasonable? }Skip

  7. Sheer number of systems • 130 systems simulated on 5 machines • 40,000 DPEs • ~5 million DPEs • Interconnected successfully

  8. What were the potential showstoppers? • Basic functionality • Synchronised types in V2! • Sheer number of systems • Can the implementation cope? • Alert Avalanches • How does PVSS degrade? • Is load of many Alerts reasonable? • Is load of many Trends reasonable? }Skip

  9. UI UI Alert Avalanche Configuration 91 94 • 2 WXP machines • Each machine = 1 system • Each system has 5 crates declared x 256 channels x 2 alerts in each channel (“voltage” and “current”) • 40,000 DPEs total in each system • Each system showed alerts from both systems

  10. Traffic & Alert Generation • Simple UI script • Repeat • Delay D mS • Change N DPEs • Traffic rate D \ N • Bursts. • Not changes/sec. • Option provoke alerts

  11. Alert Avalanche Test Results - I • You can select which system’s alerts you wish to view • UI caches ALL alerts from ALL selected systems. • Needs sufficient RAM! (5,000 CAME + 5,000 WENT alerts needed 80Mb) • Screen update is CPU hungry and an avalanche takes time(!) • 30 sec for 10,000 lines.

  12. Alert Avalanche Test Results - II • Too many alerts -> progressive degradation • 1) Screen update suspended • Message shown • 2) Evasive Action. Event Manager eventually cuts the connection to the UI; UI suicides. • EM correctly processed ALL alerts and LOST NO DATA.

  13. Alert Avalanche Test Results - III • Alert screen update is CPU intensive • Scattered alert screens behave the same as local ones. (TCP) • “Went” alerts that are acknowledged on one alert screen disappear from the other alert screens, as expected. • Bugs we reported have now been fixed.

  14. What were the potential showstoppers? • Basic functionality • Synchronised types in V2! • Sheer number of systems • Can the implementation cope? • Alert Avalanches • How does PVSS degrade? • Is load of many Alerts reasonable? • Is load of many Trends reasonable?

  15. W W L W L L L L L L L L L L L L Agreed Realistic Configuration • 3 level hierarchy of machines • Only ancestral connections, no peer links. Only direct connections allowed. • 40,000 DPEs in each system, 1 sys per machine • Mixed platform (W=Windows, L=Linux)

  16. 91 92 93 94 95 03 04 05 06 07 08 09 10 11 12 13 Viewing Alerts coming from leaf systems • 1,000 “came” alerts generated on PC94 took 15 sec to be absorbed by PC91. All 4(2) CPUs in PC91 shouldered the load. • Additional alerts then fed from PC93 to the top node. • Same graceful degradation and evasive action seen as before. PC91’s EM killed PC91’s Alert Screen • Display is again the bottleneck.

  17. Rate supportable from 2 systems 91 • Set up a high, but supportable rate of traffic (10,000 \ 1,000) on each of PC93 and PC94, feeding PC91. • PC93 itself was almost saturated, but PC91 coped (~200 alerts/sec average, dual CPU) 92 93 94 95 03 04 05 06 07 08 09 10 11 12 13

  18. 91 92 93 94 Surprise Overload (manual) • Manually stop PC93 • PC91 pops up a message  • Manually restart PC93 • Rush of traffic to PC91 caused PC93 to overload • PC93’s EM killed PC93’s DistM • PC91 pops up a message  95 03 04 05 06 07 08 09 10 11 12 13

  19. PVSS Self-healing property • PVSS self-healing algorithm • Pmon on PC93 restarts PC93’s DistM

  20. Remarks • Evasive action taken by EM, cutting connection, is very good. Localises problems, keeping the overall system intact. • Self-healing action is very good. Automatic restart of dead managers • BUT…

  21. 91 92 93 94 Evasive action and Self-healing • Manually stop PC93 • PC91 pops up a message • Manually restart PC93 • Rush of traffic to PC91 causes PC93 to overload • PC93’s EM killed PC93’s DistM • PC91 pops up a message • Pmon restarts PC93’s DistM

  22. Self-healing Improvement • To avoid the infinite loop, ETM’s Pmon eventually gives up.  • Configurable how soon  • Still not ideal! • ETM are currently considering my suggestion for improvement: • Pmon should issue the restart, but not immediately.

  23. (Old) Alert Screen • We fed back many problems with the Alert Screen during the pre-release trials. • E.g. leaves stale information on-screen when systems leave and come back.

  24. New Alert/Event Screen in V3.0 • 3.0Official release now has a completely new Alert/Event Screen which fixes most of the problems. • It’s new and still has some bugs, but the ones we have seen are neither design problems nor showstoppers.

  25. More work for ETM: • When DistM is killed by EM taking evasive action, the only indication is in the log. • But Log viewer, like Alert viewer, is heavy on CPU and shouldn’t be left running when it’s not needed.

  26. Reconnection Behaviour • No gaps in the Alert archive of the machine that isolated itself by taking evasive action. No data was lost. • It takes about 20 sec for 2 newly restarted Distribution Managers to get back in contact. • Existing (new-style!) alert screens are updated with the alerts of new systems that join (or re-join) the cluster.

  27. Is load of many Alerts reasonable? • ~200 alerts/sec average would be rather worrying in a production system. So I believe “Yes”. • The response to an overload is very good. Though can still be tweaked. • Data integrity is preserved throughout.

  28. What were the potential showstoppers? • Basic functionality • Synchronised types in V2! • Sheer number of systems • Can the implementation cope? • Alert Avalanches • How does PVSS degrade? • Is load of many Alerts reasonable? • Is load of many Trends reasonable? 

  29. Can you see the baby?

  30. What were the potential showstoppers? • Basic functionality • Synchronised types in V2! • Sheer number of systems • Can the implementation cope? • Alert Avalanches • How does PVSS degrade? • Is load of many Alerts reasonable? • Is load of many Trends reasonable?

  31. Is the load of many Trends reasonable? • Same configuration: 91 92 93 94 95 03 04 05 06 07 08 09 10 11 12 13 • Trend windows were opened on PC91 displaying data from more and more systems. Mixed platform.

  32. Is Memory Usage Reasonable? Yes

  33. Is Memory Usage Reasonable? Yes

  34. Test 34: Looked at top node plotting data from leaf machines’ archives • Performed excellently. • Test ceased when we ran out of screen real estate to show even the iconised trends (48 of).

  35. Bland result? No! • Did the tests go smoothly? No! • But there was good news at the end 

  36. Zzzzzzz Observed gaps in the trend!! • Investigation showed gap was correct  • Remote Desktop start-up caused CPU load • Data changes were not generated at this time

  37. Zzzzzzz Proof with a Scattered Generator Trend UI on PC94 • Steady traffic generation • No gaps in the recorded archive • Even when deliberately soak up CPU • Gaps were seen in the display • Need a “Trend Refresh” button (ETM) Scattered UI on PC93 Traffic EM

  38. Zzzzzzz Would sustained overload give trend problems? • High traffic (400mS delay\1000 changes) on PC93, as a scattered member of PC94’s system. • PC94’s own trend plot could not keep up. • PC91’s trend plot could not keep up. • “Not keep up” means…

  39. Zzzzzzz “Display can’t keep up” means… Timenow Trend screen values updated to here

  40. Zzzzzzz Evasive action EM took evasive action, (disconnected the traffic generator) just here Timenow Trend screen values finally updated to here Last 65sec queued in Traffic Generator. Lost when it suicided.

  41. Wakey! Summary of Multiple Trending • PVSS can cope • PVSS is very resilient to overload • Successful tests.

  42. Test 31 DP change rates • Measured saturation rates on different platform configurations. • No surprises. Faster machines with more memory are better. Linux is better than Windows. • Numbers on the Web.

  43. Test 32 DP changes with alerts • Measured saturation rates; no surprises again. • Dual CPU can help in processing when there are a lot of alert screen (user interface) updates.

  44. What were the potential showstoppers? • Basic functionality • Synchronised types in V2! • Sheer number of systems • Can the implementation cope? • Alert Avalanches • How does PVSS degrade? • Is load of many Alerts reasonable? • Is load of many Trendsreasonable? • Conclusions

  45. Conclusions • No showstoppers.   • We have seen nothing to suggest that PVSS cannot be used to build a very big system.

  46. Further work - I • Further “informational” tests will be conducted to assist in making configuration recommendations, eg understanding the configurability of the message queuing and evasive action mechanism. • Follow up issues such as “AES needed more CPU when scattered”. • Traffic overload from a SIM driver rather than a UI • Collaborate with Peter C. to perform network overload tests.

  47. Further work – II • Request a Use Case from experiments for a non-stressed configuration: • Realistic sustained alert rates • Realistic peak alert rate + realistic duration • i.e. not a sustained avalanche • How many users connected to control room machine? • % viewing alerts; % viewing trends; % viewing numbers (eg CAEN voltages) • Terminal Server UI connections • How many UIs can control room cope with? • What recommendations do you want?

  48. In greater detail… • The numbers behind these slides will soon be available on the Web at http://itcobe.web.cern.ch/itcobe/Projects/ScalingUpPVSS/welcome.html • Any questions?

  49. Can you see the baby?

More Related