270 likes | 502 Views
Unity Connection 2.0 Mixer. UCBU Media Team June 2007. Agenda. Overview of Mixer Architecture and Features Placement of mixer in Unity Connection 2.0 architecture Mixer characteristics / supported features Administrative Interface Control Interface and Signaling
E N D
Unity Connection 2.0 Mixer UCBU Media Team June 2007
Agenda • Overview of Mixer Architecture and Features • Placement of mixer in Unity Connection 2.0 architecture • Mixer characteristics / supported features • Administrative Interface • Control Interface and Signaling • API used for Miu to CuMixer messaging • CuMixer communicates with mixer via API library calls
Agenda (continued) • Debugging • Tracing mixer API requests and responses • Logging of debug and error messages • Interactive Diagnostic Interface (TAC CLI) • Troubleshooting audio problems
Mixer Overview • Mixer features: • Mixer is software only, no DSPs are used. • It runs as a user mode process (not a kernel driver). • Supports G.711(u/a), G.726 (32kbps), and G.729a codecs (all mono, 8K sample rate) • Voice Activity Detection (VAD, with comfort noise packet sent at start of silent periods) • RFC-2833 DTMF relay (both send and receive) • Performs Text To Speech (TTS) playback of text files using Nuance RealSpeak • ICMP port unreachable detection and reporting for media failure • Maintains numerous statistics on channels
Mixer placement in Unity Connection 2.0 CsMgr TAC-CLI (Diag. Interface) GY_API Message Queue CuMixer Control Application (process) API CuMixer makes function calls to the mixer API layer Soft-mixer (gy_mixer) components File I/O Player - Recorder Nuance TTS Engine Audio Mixing RTP Logging • Decode/encode RTP streams • “fork” audio from EP to ASR • Recording from endpoints to disk • Playback of audio and text files. Log Files for debugging
Mixer Debugging • Options Accessible to Administrator • Mixer Log Files: • Mixer log files are found by using Real-Time Monitoring Tool (RTMT) “Trace & Log Central” to collect the mixer log files. • Log contains error information and event messages, detail depends on selected level. • Debugging level can be set using the Unity Connection Serviceability web page: • Go to Unity Connection Serviceability->traces->micro traces->mixer • Set level to value between 0 and 6 (6 is most detailed debugging traces) • Or, go to Unity Connection Serviceability->traces->macro traces->media (wave) traces, which turns on Miu and Mixer tracing. • Error conditions are always logged, regardless of logging level. (Default debugging level = 4.) • Log file name: • Filenames are “diag_CuMixer_00000xx.uc” where xx=0 thru 39. • Log files close after 200K written, and next file is opened. • Log files wrap after 40 files.
Debugging Mixer Issues Checking the mixer logs for errors: 1. search logs for keywords “failure” or “error”: -- You may see ICMP port unreachable log msgs. These are reported when the mixer is streaming to an endpoint that has closed its RTP port. It usually means some sort of signaling problem, where the call has terminated but the CUC session has not. -- Any other instance of “error” or “failure” is significant, and could indicate: a cmd sequence error or invalid request received; Bad recorded prompt, or other issue. 2. Turn on more detailed logging if needed.
Debugging Mixer Issues • Search logs for API message types to follow progress of call. • Some Useful Message Types: • GY_MSG_OPEN_CHN: Start of an endpoint sessionGY_MSG_CLOSE_CHN: End of an endpoint sessionGY_MSG_OPEN_FORKGROUP: Create a new fork groupGY_MSG_CLOSE_FORKROUP: Close a for groupGY_MSG_ADD_CHN_FORKGROUP: Add a channel to a FGGY_MSG_RM_CHN_FORKGROUP: Remove a channel from a FGGY_MSG_PLAY_FILE: Play an audio or text fileGY_MSG_STOP_PLAY_FILE: Pause playback of a fileGY_MSG_CLOSE_PLAY_FILE: Terminate and close playback of a fileGY_MSG_PLAY_FILE_EVT: From mixer – indicates play file doneGY_MSG_RECORD_FILE: Start recording to a fileGY_MSG_STOP_RECORD_FILE: Pause file recordingGY_MSG_CLOSE_RECORD_FILE: Stop recording and close file • NOTE: Mixer response messages to the Miu commands above have same mnemonic, but with “_RES” appended to the end of the message ID.
Mixer Debugging • Options Accessible with “root” CLI Access • Mixer Log Files: • Mixer log files can also be accessed directly in directory “/var/log/gy” when logging into the server as “root” and using “cd” to navigate to that directory. • Filenames are “cisco_gy_log_file_xx” where xx=0 thru 39. • :
TAC Command Line Interface • Text-based command line interface “taccli”: - “root” logon required. Just type “taccli” to start it up. - Provides access to mixer diagnostic information - Allows control of mixer tracing level (alternative to CUCS screen) - Display GYAPI messaging counters, errors, other stats - Activate / deactivate channel diagnostic recording - Display mixer return code table for error code lookup - Display other information
TAC Command Line Interface • Menu Screen: • [root@unityconnection root]# taccli • CLI to mixer queue created, Queue ID=0x12340001 • ********************************************************* • Unity Connection Mixer Diagnostics Interface • ********************************************************* • Please enter a request: • 1 Display statistics for all channels 8 Activate tracing • 2 Display statistics for a channel 9 Deactivate tracing • 3 Display statistics for all fork groups 10 Show current date/time • 4 Display fork group statistics 11 Display status of a channel • 5 Display mixer status 12 Display mixer return code table • 6 Display fork group details 13 Start/Stop channel RTP recording • 7 Quit • :
TAC Command Line Interface All Channels statistics display (option 1) Displaying statistics for all channels: ******************************* - 118 - **************************************** Chn ID: 118 Start Time: 05/01/07 16:37:34 Elapsed Time: 0:00:41 Total Pkts Recv: 0 Total Pkts Sent: 2 Total Bytes Recv: 0 Total Bytes Sent: 161 Total Pkts lost/Too late to play: 0 Packet Size: 0 Max. Jitter: 0 Endpoint IP Addr: 172.19.248.66 Endpoint RTP Port: 7774 ************************* - End of channel display - *************************
TAC Command Line Interface All Channels statistics display (option 1) continued ******************************* - 119 - **************************************** Chn ID: 119 Start Time: 05/01/07 16:37:22 Elapsed Time: 0:00:53 Total Pkts Recv: 425 Total Pkts Sent: 1387 Total Bytes Recv: 60527 Total Bytes Sent: 218853 Total Pkts lost/Too late to play: 1 Packet Size: 160 Max. Jitter: 1 Endpoint IP Addr: 10.32.132.52 Endpoint RTP Port: 28822 ************************* - End of channel display - *************************
TAC Command Line Interface Single Channel Statistics Display (option 2) ******************************* - 119 - *********************************** Chn ID: 119 Start Time: 11/27/06 21:30:29 Elapsed Time: 0:00:03 Total Pkts Recv: 78 Total Pkts Sent: 89 Total Bytes Recv: 12480 Total Bytes Sent: 14240 Total Pkts lost/Too late to play: 0 Packet Size: 160 Max. Jitter: 4 Endpoint IP Addr: 10.21.145.2 Endpoint RTP Port: 49152
TAC Command Line Interface Channel Statistics field explanations Chn ID: Channel number Start Time: Time at which the channel was opened Elapsed Time: Amount of time the channel has been open (in seconds) Total Pkts Recv: Total packets received from the channel endpoint Total Pkts Sent: Total packets sent to the channel endpoint Total Bytes Recv: Total bytes received from the channel endpoint Total Bytes Sent: Total bytes sent to the channel endpoint Total Pkts lost/Too late to play: Total dropped packets Packet Size: in bytes Max. Jitter: in millisec. This is the maximum observed value of the average pkt jitter Endpoint IP Addr: Remote IP address of the endpoint Endpoint RTP Port: Remote RTP port used by the endpoint
TAC Command Line Interface All Forkgroups Display (Option 3) Display all forkgroups: ******************************* - 249 - **************************************** ForkGroup ID: 249 ForkGroup Sched. ID: 0 Start Time: 05/01/07 17:09:18 Elapsed Time: 0:02:00 Num. Active Chn: 0 Max. Active Chn: 2 Join/Depart ForkGroup: 16 Total Pkts Recv: 209 Total Pkts Sent: 1737 Total Bytes Recv: 18971 Total Bytes Sent: 275694 Max. Jitter: 1 Max. Playout Delay: 0 Last Playout Delay: 0 Max. Record Delay: 0 Last Record Delay: 0 *********************** - End of ForkGroup Display - **********************
TAC Command Line Interface Forkgroup Statistics Display (Option 4) Display forkgroup statistics: Please enter a forkgroup ID between 0 and 249: 249 ******************************* - 249 - **************************************** ForkGroup ID: 249 ForkGroup Sched. ID: 0 Start Time: 05/03/07 14:14:30 Elapsed Time: 0:00:33 Num. Active Chn: 2 Max. Active Chn: 2 Join/Depart ForkGroup: 14 Total Pkts Recv: 136 Total Pkts Sent: 84 Total Bytes Recv: 9517 Total Bytes Sent: 13122 Max. Jitter: 2 Max. Playout Delay: 0 Last Playout Delay: 0 Max. Record Delay: 0 Last Record Delay: 0
TAC Command Line Interface System-wide Statistics Display (Option 5) ** Unity Connection Mixer Systemwide Statistics ** Mixer Start Time: 05/01/07 16:35:24 Elapsed Time: 0:48:44 ForkGroup Active: 1 Max. Simultaneous ForkGroup: 1 ForkGroup Total (cumulative): 4 Current Active Channels: 2 Channels Total (cumulative): 8 Longest ForkGroup Duration: 128 Total Packets Received: 2709 Total Packets Sent: 14060 Total Bytes Recv: 357438 Total Bytes Sent: 2232905 Total Packets Dropped: 18 Max. Jitter Observed: 2
TAC Command Line Interface ForkGroup Details Display (Option 6) Please enter a forkgroup ID between 0 and 250: 249 ******************************* - ForkGroup Status - ******************************* forkgroup ID: 249 ForkGroup Name: conf249 ForkGroup State: Open Currently playing prompt: None Current record file: None Number of ForkGroup channels: 2 ForkGroup channel IDs: 118 119 Number of contributing channels: 1 Channel IDs: 119
Activate/Deactivate Tracing; Show Date-Time (Options 8, 9, 10) : 8 Request: Activate Tracing: Trace levels are 0-6 (6=Maximum). Enter desired trace level: 6 selection was: '6' Mixer trace activated. : 9 Request: Deactivate Tracing: Trace deactivated. :10 Current system time: 05/03/07 14:25:03
TAC Command Line Interface Channel Status Display (Option 11) ******************************* - 119 - **************************************** Chn. ID: 119 Channel Status: Active, not currently in conference Conf. ID: None Contribute to mixer: No Endpoint IP Addr: 10.21.145.2 Endpoint RTP Port: 49152 Endpoint RTCP Port: 0 Codec Type: G711aLaw Ingress frame Size: 20 Bitrate: 64000 Ingress Payload Type: 8 Ingress DTMF Payload Type: 101 Ingress DTMF Enabled: Yes Mixer IP Address: 172.19.136.149 Mixer RTP Port: 19794 Mixer RTCP Port: 19795 IP TOS Precedence: 0x 2e TTL: 64 Egress Codec Type: G711aLaw Egress Frame Size: 20 Egress Bitrate: 8000 Egress Payload Type: 8 Egress DTMF Payload Type: 101 Egress DTMF Enabled: Yes Egress Mute: No Egress VAD: No Prompt Active: Yes Prompt File Name: /opt/cisco/meetingplace_express/afs//runprompt/en_US/s1739.wav Record Active: No Diagnostic Record Active: No
TAC Command Line Interface Start/Stop Channel RTP Recording (Option 13) :13 Please enter a channel ID between 0 and 250. (To deactivate all diag. recording, enter 999): 118 Enter 1 to start or 0 to stop recording: 1 Recording activated, saving to file /tmp/chn.118.05.07.07.16.10.03.wav :13 Please enter a channel ID between 0 and 250. (To deactivate all diag. recording, enter 999): 118 Enter 1 to start or 0 to stop recording: 0 Recording deactivated for RTP stream 118.
Debugging Audio Quality Issues • Debugging audio stream problems: • Locate the channel of interest, and check the mixer and • channel statistics for excessive jitter. • Options if the problem can be isolated to a particular caller: • Use diagnostic channel recording to capture what is coming into the mixer. • Check channel stats for dropped packets, jitter, etc. • Try to see if the audio was already bad coming into the mixer: • Use Ethereal to capture audio in and out of the mixer (eth0). Ethereal can • create an “.au” file from G711 RTP, and do stream analysis. • -- Select any RTP packet from the problem endpoint • -- Select: “Statistics”->”RTP”->”Stream Analysis”->”Save Payload” • -- Select “forward channel”, then specify file name (c:\xxx.au), then save file. • -- The .au file can then be played with Windows media player. Is the incoming audio • stream already bad? If so, then look for network problems.
Debugging TTS Issues • The TTS Engine is a third-party product from Nuance Communications, Inc. • From Cisco Unity Connection Servicabilty web page: • Go to CUCS traces->macro traces->Text to Speech traces • Turn on tracing • Use RTMT to collect traces and search for “error” • Or, with root access: • Edit file /usr/local/Scansoft/RealSpeak4.0/config/ttsserver.xml. • Make sure parameter “log_file_enabled” is set to “true”. • Parameter “log_file_base_name” is the name of the log file • Search for “error” in the log file (XML format)
Currently open bugs The following are known mixer issues which are expected to be fixed in the next release: • CSCsi39808: VUI voice commands may occasionally not be recognized by the mixer’s ASR engine. This problem is under investigation. • CSCsi41515: Some .wav prompt files may contain other chunks preceding the “fmt “ chunk, and the mixer will not be able to play such .wav files. This would only be seen if a user has substituted their own prompt files for the standard ones provided, and if such provided files have this unusual characteristic. This issue is fixed in the next release. • CSCsi60118: If a user attempts to use the TTS feature with a language which is not installed on that system, they may experience a delay without any playback, any error response, or other feedback, until the maximum time-out for playback occurs. This is fixed in the next release. • CSCsj14600: Comfort Noise packets not using industry standard payload type value.
For More Information • Related Docs: • Lucas Messaging Software Architecture Proposal, EDCS-490444 • Lucas: Mixer and App Interfaces, EDCS-500086 • Bounty Hunter core audio mixer SUFS/SUDS – EDCS-415152 • Bounty Hunter core mixer diagrams: EDCS - 486998