Intermittent Static on Unity TRaP Port on Nortel Integration

schonb01 · ‎06-10-2008

I have a customer that has intermittent issues on the first TRaP port

(port 40) in a Nortel Option 81C digital PIMG integration to Unity 5.0(1). The PIMG makes the call to the phone, the person answers, and all they hear is static. The port just has to be unplugged and plugged back in to reset it and then it will work again for about

another month. This is pretty unacceptable to them, as you might imagine, since Unity doesn't see this as a failed call and continues to use this port for TRaP calls. We disabled the TRaP ports on this PIMG unit (PIMG5) and ran off PIMG4 (next up in the stack) for awhile and the same thing happened, so it is not localized to just PIMG5.

Does anyone have any feedback regarding:

1) What are the recommened PIMG traces (I would imaging Tel and VoIP) and Unity traces that I need to enable when the problem reoccurs in order for TAC to find some answers.

2) The recommended PIMG code release?

3) Similar issues? Possible solutions?

Stack of five UNITY-PIMG-DIG (Dialogic DMG1008DNIW) Units

Nortel Meridian Option 81C

PIMG Software 5.1 SU2

Unity 5.0(1) ES33

eschulz · ‎06-11-2008

Since the problem seems to be tied to a specific port, I'd suspect the issue lies on PIMG, Nortel or in between. But to be sure, you could grab a network capture between Unity and PIMG during the next occurrence. In the capture you should be able to verify two things: 1) audio sent from Unity is clear, not static 2) codec used in the RTP stream matches what was negotiated in the SIP signaling. If you find a problem in that network capture, then you should turn you focus to the Unity diags.

For the PIMG-Nortel side, the Tel and VoIP traces sound like a good starting point. I think maybe there are some DSP related traces available too. Does the digital PIMG also allow you to capture the audio from the TDM side? If so, that might be worth getting.

For your second question, the latest PIMG software recommended by Cisco is available on Cisco.com. Looks like we're currently at 5.1_SU3 but I think a 6.0 release will be coming out soon. Here's an ugly but direct link...

http://tools.cisco.com/support/downloads/go/Model.x?mdfid=282021185&mdfLevel=Software%20Version/Option&treeName=Voice%20and%20Unified%20Communications&modelName=Cisco%20Unity%20Telephony%20Integration&treeMdfId=278875240

For your third question, I haven't seen this before. If no one else here has then I'd strongly recommend working with Cisco TAC. They should be able to help with getting the right traces and interpreting them.

Regards,

Eric

schonb01 · ‎06-11-2008

We were able to gather Tel and VOIP traces this morning when we had the issue, but at that point, I hadn't read this posting yet (so no network capture). I couldn't see anything particular wrong in those traces (not that I can decode everything there). I submitted it to TAC and also attached it if anyone is interested.

TAC also came back and said that there have been static issues resolved with SIP integrations of Unity, such as with PIMG, since Unity 5.0(1) was released (e.g. CSCsm73465). They recommended installing ES44 as a first step. We will be installing it probably tonight if the customer gives me an outage window.

Great idea on the network captures. I did some test captures from the PIMG today for working TRaP calls. Wireshark (formerly Ethereal) allows you to look at the SIP call (Statistics-->VoIP Calls), decode it, and play it in the built-in player. I attached a sample file of the good call, if you or anyone are interested in looking at it. I also submitted the idea in my TAC case. There is no way to get the audio trace from the TDM side, but the customer is looking at installing some kind of inline tap that would allow us to play the call over a normal audio device. It would at least allow us to see if the PIMG is sending good audio to the PBX (as long as Unity is sending good audio as well to the PIMG and there is no issue there).

To my understanding, the problem has occurred every morning for the last three mornings, but before that, we had only three other occurrances, each about a month apart (PIMG5, port 40 twice and PIMG4 once) and now PIMG4 on Monday, PIMG4 on tuesday, and PIMG5 today, since we re-enabled the 2 disabled TRaP ports on that box. There is a nightly maintenance process that checks station ports and disables them if no phones are connected, so we will look more closely at that, since the problem seems to happen in the morning. I will be also testing before and after the PBX maintenance process runs and before and after the MWI refresh at 2 am, to see if there is any correlation.

Thanks for the tips so far!

eschulz · ‎06-11-2008

Well, the audio from Unity in the good call capture sounds fine. ;) Now you just need to wait and capture a bad call for comparison.

CSCsm73465 was tied to Unity's decrypting of secure messages. That shouldn't impact playback of a recorded name or greeting via TRaP. Getting a recent ES is not a bad idea though.

In general, I doubt any changes to the Unity code will fix this. You state that the problem can be resolved simply by resetting the TDM line plugged into the PIMG port. Unity of course has no visibility into when this happens. So if it were a Unity bug, I don't see how resetting the TDM port would help. Okay, I suppose something funny could be happening between Unity and PIMG that puts the TDM side of the PIMG port into a bad state such that subsequent calls get static until the TDM side is reset. That's kind of a stretch though and there certainly haven't been any recent bug fixes along those lines.

Getting a capture from the TDM side would be nice, but with a digital PIMG I don't know how they expect to tap the line.

The nightly maintenance is definitely something worth looking into. I haven't heard of it causing static issues but it certainly has bitten folks in other ways.

One more thing to try next time you have a port in the bad state... can you place an inbound call through that port? If so, does the same static occur or is the static only present when Unity initiates the call?

schonb01 · ‎06-11-2008

Thanks, Eric. I reviewed the bug ID as well and I'm also of the opinion that it probably isn't Unity causing the issue, but we will know soon enough for sure by doing the network capture when the problem happens. TAC did mention static issues (more than one, but only cited the secure messaging bug), but looking at the list of bug fixes on ES44, none of them seemed to be obviously related to static issues. Having the ES on there will also help placate TAC a little bit and it certainly (usually) doesn't hurt.

We'll keep looking at the maintenance processes and also look at inbound calls to the port when it happens (good suggestion!).

Thanks again for the replies!

schonb01 · ‎06-12-2008

ES44 was applied on schedule last night with no issues. I also tested TRaP before and after PBX maintenance (I ran my test at about 1:30 a.m.), and I had no issues. At this point, we wait to see if the problem happens again, and if so, I will gather the network trace to look at the SIP leg of the call and work backwards towards the PBX.

schonb01 · ‎06-27-2008

Ok, as it turns out, we had the problem happen again. One thing that we do notice is that on MWI ports, the PIMGs log a lot of "TDM out of Service" messages. Any input on what to do next would be welcome. I have sent more info to TAC too. Also, I verified that what the PIMG receives from Unity is indeed a good audio stream (via a network capture).