1 / 22

Lessons Learned Enhancing EPICS CA for LANSCE Timed and Flavored Data

Lessons Learned Enhancing EPICS CA for LANSCE Timed and Flavored Data. Jeffrey O. Hill LANSCE / LANL. Outline. Requirements, a Review Design, a Review Design Changes Status, this project and others Lessons Learned Benefits, a Review. Requirements - Subscription Filtering.

terrydonnelly
Download Presentation

Lessons Learned Enhancing EPICS CA for LANSCE Timed and Flavored Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lessons Learned Enhancing EPICS CA for LANSCE Timed and Flavored Data Jeffrey O. Hill LANSCE / LANL

  2. Outline • Requirements, a Review • Design, a Review • Design Changes • Status, this project and others • Lessons Learned • Benefits, a Review

  3. Requirements - Subscription Filtering • LANSCE timing and flavoring of data • Flavoring • Selection based on - logical combinatorial of beam gates • Timing • Selection based on - time sampling within beam pulse • Many permutations • Too many to, a-priori, install records for all of them • Client side tools specify flavoring when subscribing • Capabilities to forward timing and beam gate companion data in IOC core • Filtering capabilities in the CA server

  4. Requirements - Time Metadata • LANSCE timed data requires Array Index Metadata • Magnitude of zero-eth element index • Floating point • Magnitude of one index increment • Floating point • Units of these magnitudes • String

  5. Design - Generic Implementation • Generic interface to site specific subscription update payloads • Data Access • What it isn't • A storage format or a linked list of discriminated unions ala GDD and CDevData • We have done this before - there is a better way IMHO • What it is • A generic interface for introspecting hierarchical data • Data is interfaced in their native (site specific) data structures • A more general approach • We can wrap GDD with DA, but the converse isn't possible • Smart pointer references to site specific payloads • Reference counting • Efficient memory management in device or record support • Mutual exclusion, or not, determined by device or record support • Filtering expressions specified as strings • Minimal impact to legacy client side tools

  6. Design - Event Queue Upgraded Record Specific Event Parameter Set Scalar or Vector Value Time Stamp Alarm Status … Upgraded Device Specific Event Parameter Set LANSCE Beam Gate Specification Legacy Fixed Event Parameter Set Scalar Value Time Stamp Alarm Status Record CA Server Event Queue

  7. TCP Circuit Design Event Filter Record Event Filter Event Queue Event Filter CALC Expression EX1 beamGateSpec & ( beamGateA | beamGateB ) && ! (beamGateSpec & beamGateC ) EX2 Value > 100 Discard

  8. Design Changes - MutalExlusion • Before – Guard based Mutex mutex; Guard guard (mutex ); Service & service = createService (guard); Channel & channel = Service.createChannel (guard,…); • After – smart pointer based Ptr < Service > pService = createService (); Ptr < Channel > pChannel = pService->createChannel (…);

  9. Design Changes – Smart Pointers • For event queue implementation and public CA interfaces we have some special requirements – for flexibility and isolation • Data Access Catalog, CA Channel, and others are used by different threads than their creators • Runtime polymorphism – requires virtual interface • User of Ptr must not know at runtime or compile time which of these are chosen • Reference Counted memory management, or not • Mutual Exclusion protection for target, or not • Intrusive (i.e. within target object) implementation, or not

  10. Design Changes – Smart Pointers • Surveyed popular smart pointers in particular • Boost shared_ptr • Reference counting - always • No mutual exclusion for target • only for reference count • Non-intrusive implementation – always • Can be less efficient (locked allocation of reference counter) but better code reuse • Modern C++ Design smart pointer (Alexandrescu) • The ultimate in terms of flexibility, and efficiency potential • Complex template parameter based policy selection • Pushing the bleeding edge of compiler feature sets • Policy selection template parameters can be verbose, intimidating for casual user • this is easily ameliorated by • typedefing runtime polymorphic policy selection • Implement specializations for runtime polymorphic behavior

  11. Design Changes • After some deliberation a custom implementation appeared to be the best choice • We need mutual exclusion which boost library’s shared_ptr does not provide • The loki library’s smart pointer extensive use of templates is probably not portable enough for EPICS. Legacy vxWorks systems use an old gnu compiler. • A custom implementation currently requires 547 lines of code – not a significant maintenance burden

  12. Design Changes – Smart Pointers • Internally a smart pointer has a single data member • Pointer to a handle • Smart pointer same size as ordinary pointer • A handle is runtime polymorphic • The operator-> is overloaded to return locked smart pointer object class HandleIntf { virtual TargetMutexPair targetMutexPair () = 0; virtual HandleIntf & clone () = 0; virtual void release () throw () = 0; };

  13. New Atomic Primitives in R3.15 • Efficient atomic increment / decrement needed for reference counting • RISC architecture requires mutual exclusion • Increment requires two (interruptible) instructions • SMP architecture requires memory barriers • Cache sync / flush operations • SMP, the future CPU architecture • Atomic increment performance - versus a mutex • On typical computer (Intel core2duo 3GHz) • 70 μS epicsMutex lock/unlock • 23 μS osiAtomicIncr • Increment / Decrement Implementations (so far) • GCC 4.1 intrinsic primitives on OS/GNU • OS primitives on Solaris, Windows, vxWorks • OS, predating SMP, lock interrupts very briefly • Legacy vxWorks, RTEMS, …

  14. Status, MAGVIZ, my Full Time Project Prev Year + 2 Months Before • MagViz distinguishes potential-threat liquids from harmless shampoos and sodas screened at airport baggage checkpoints

  15. Status, EPICS R3.14.11 • Many quality improvement related changes in the portable server library • Fixed lockup when flow control started and channel connect requests were pending on the event queue • This may be the cause of DB CA links not reconnecting through the gateway at Diamond and SLS • Incoming (put) data corruption under heavy CAS load. 2nd occurrence found • Faster string conversions - delays reduced, large arrays • Scheduling improvements, large arrays, many others … • The portable server library is used by the CA gateway, and application level services

  16. Status, CA Gateway Changes • CA Gateway was modified to behave differently for ca put versus ca put callback • CA put • Caching behavior when client put production rate exceeds put consumption rate of IOC • Classic MEDM slider over-pumps the slow motor record situation – is handled well now • CA put callback • The original behavior is preserved • Each and every put request is executed

  17. Status, This Project • I am working again on this project since the start of the calendar year • Some interruptions this summer due to EPICS R3.14.11, CA gateway changes • Guard to smart pointer locking model transition required extensive reorganization of the data structures, but is now close to completion, and worth the delay IMHO • The code builds, but there are currently about 40 undefined symbols

  18. Lessons Learned • Multithreaded and Object Oriented • Designing for both together takes time to master • Changing the primary interface’s locking model during the middle of the project can be like pulling on a sweater thread. • When designing any modern code SMP must be considered • Some additional locking overhead is justified when we consider that N processors can work concurrently • Nevertheless, I do worry about the cost of blocking processors while they reload their pipeline due to memory barriers (lesson is pending)

  19. Lessons Learned • Working on multiple project simultaneously • Positive impacts • Returning to a project leads to a fresh object and a desire to review core design decisions • Negative impacts • Too much context switching • Lost momentum

  20. Lessons Learned • Breaking large projects up into smaller upgrades is a good idea • Incremental upgrades occur more quickly • With more frequent releases means better feedback • More authors can share the development load

  21. Benefits for LANSCE • LANSCE style dynamic on-the-fly ad-hoc beam flavoring and beam timing experiments • But, in homogenous EPICS system • Tool based approach to LANSCE applications • Applications have abstract model of hardware • Incremental upgrades hopefully easier • Homogeneous communication model for high level apps might make on-call duties easier • Multi-element “Timed” data • COTS digitizer • Window in time selected

  22. Benefits for EPICS Community • Flexible event snapshots • Parameters other than alarm status, time stamp, scalar value correlated on event queue • Array update bursts buffered on event queue • Subscription update filtering • Expression (string) based means • Project and site independent – tool based approach • Minimally invasive for existing client side tools • Array index meta data • Expanding intersection of EPICS with data acquisition systems

More Related