220 likes | 553 Views
Tango polling system. Tango workshop Archamps (08/10/2005). Introduction to Tango polling. Every Tango device server process has a polling thread Its rule is to Poll attribute and/or command (without input parameters)
E N D
Tango polling system Tango workshop Archamps (08/10/2005)
Introduction to Tango polling • Every Tango device server process has a polling thread • Its rule is to • Poll attribute and/or command (without input parameters) • Store result in a data cache (called polling buffer) managed as a circular buffer • Detect events and throw them to the CORBA notification service
Introduction to Tango polling • Each device has its own polling buffer • Polling buffer depth is tunable • By device (default is 10) • By command/attribute for finer tuning • Client get their data from • The real device • The last record in the polling buffer • The polling buffer and in case of error from the real device • The choice is done by a Tango API client call
Introduction to Tango polling • Configuring polling means • Defining which command/attribute must be polled • Defining polling periods • Every polled object (command or attribute) has its own polling period • Two way to configure polling • From the Tango database (using Jive) • Directly in code (the way used by Pogo)
Introduction to Tango polling • According to the way polling is configured (db or code), polling starts • At device server process start-up time if configured from database • At the first request to read data from polling buffer when configured by code
The Polling has to be tuned • Nothing is magic! • Don’t believe that the polling thread is able to poll everything at high frequency simply by decreasing polling period parameter • The hardware access time has to be taken into account • If you poll 5 objects (command or attribute) with 100 ms response time each, don’t try to poll them faster than 500 ms…(That’s life…)
Polling tuning rule • Reserve some time for clients other than the polling thread to access your device • The polling thread should not access the device for more than 50/60 % of time • In the previous example, set polling period at 1 second leaving device free for external client during 50 % of time
Checking polling status • The Tango device server process administration device has one command to check polling • Its name is DevPollStatus • This command returns • Polling buffer depth • Polling period • Time needed to execute last command or to read attribute • Delta time between the last recorded data (to check if it follows polling period) • Jive displays the result of this command
De-tuned polling • It may happen that the hardware response time dramatically increase (disconnected serial line..) • The polling thread is out of synchronization • It tries to return on time by DISCARDING some work. Att 1 reading time Time Att 1 Att 2 Att 3 Att 4 Att 2 and Att 3 WILL NOT be polled !!
De-tuned polling • Without discarding some works, we could saturate the device preventing any device access for the external world Att 1 reading time Att 2 reading time Att 3 reading time Time Att 1 Att 2 Att 3 Att 4 The device is not accessible any more for external world
De-tuned polling • It could even be worse: • The polling thread is out of sync. • Several polled objects are not polled any more • The data stored within the polling buffer get older and older • They even become too old (4 times the polling period) to be returned by the API. The API generates device access to return data to client instead of getting data from polling buffer • We generate additional hardware access !!
Tango kernel polling tuning • Tango kernel tries to automatically tunes the polling thread • At device server process startup time for polling configured from the database • Every 500 poll times • After a new object has been added to the polling thread list of polled object
Tango kernel polling tuning • Tuning at device server startup time • We don’t know attribute reading time nor command execution time. • The polling tops are equally spread in the smallest polling period • Ex : 3 attributes polled at 600 mS, 1000 mS and 3000 mS. The startup tuning will start polling at • T0 for attribute 1 • T0 + 200 mS (600/3) for attribute 2 • T0 + 400 mS ((600/3) * 2) for attribute 3 • This is done by a temporary thread dedicated to this issue
Tango kernel polling tuning • Regular kernel polling tuning • We now know command execution time or read attribute time • The polling tops are spread in the smallest polling period according to their previous response time
Tango kernel polling tuning • Ex : One device with 3 attributes polled • Att 1 with polling period = 600 mS needing 400 mS to be read • Att 2 with polling period = 1000 mS needing 40 mS to be read • Att 3 with polling period = 3000 mS needing 10 mS to be read • Dead time is 150 mS (600 – (400 + 40 + 10)) • Att 1 polled at T0 • Att 2 polled at T0 + 450 mS (400 + (150 / 3)) • Att 3 polled at T0 + 540 mS (400 + 40 + 2 * (150 / 3)) • Note that in this case, device is free for external access only 29% of time (10 + 3(40) + 5(400) = 2130 mS compared to 3000 mS)
Event system and polling • Event system needs polling • Don’t forget to take this into account when defining polling period for attribute/command • As soon as one application subscribes to one event on attribute, the event system ask polling thread to poll this attribute with a default polling period set to 1 sec if not already polled
Event system and de-tuned polling • Tango release 5.3 • If the polling thread is out of sync and discards some attribute reading • If one event has been subscribed for this attribute • An event with error “Out of sync” is forced informing client that polling needs tuning
Possible improvement (V 5.4 ?) • Add an attribute to device server process admin device with read value set to polling thread sleeping time during the last 5 seconds (or something like) • In case of de-tuned polling, modify automatic tuning that it does not discard always the same polled object (with permutation of the polled list object)
Possible improvement (V 5.4 ?) • Stop attribute polling for attribute polled only for event detection when no more clients are interested in event (With the help of the event heartbeat)