Re: Multiple Agent Phones ringing

texasjet79 · ‎08-05-2010

ICM 7.5.8

CAD 7.5.8

CVP 7.0.(2) with latest ES ---- With just static routes

CallManager 6-1-2.1000-13

We are seeing multiple phones ring at the same time for the same inbound caller upon Queue selection.

Has anyone else seen this and if so what was the fix?

geoff · ‎08-05-2010

Never seen this.

It's hard to imagine that ICM/CVP is at fault. Let's walk through what happens when the Router chooses the next agent from the queue to take the call.

The Router chooses an agent by SkillTargetID and asks the Peripheral for the label on the Routing Client (CVP) of the device target of the aforementioned agent. Then it returns this label to CVP. CVP looks in the static routes and sees that the destination is a subscriber, so it sends a SIP INVITE to that subscriber. It responds, and CVP tears down the VRU leg and extends the leg to the Call Manager who uses SCCP to set up the phone path.

In all of this I can't see how more than 1 label could return. I would suggest the problem is in CUCM, but I don't know exactly what is wrong. Are there pickup groups configured?

Regards,

Geoff

Anthony Holloway · ‎08-05-2010

I have no first hand experience with this, but I have heard of this happening in UCCX 7x. Two agents will receive the same inbound call, and end up on a conference with the caller.

There is currently an open TAC case, and if a resolution is provided, or defect identified, i will update this post.

david.macias · ‎08-05-2010

Is it always the same two phones? Any line apperances, hunt groups, etc?

david

Blog

texasjet79 · ‎08-05-2010

It appears to be random and not reproducable.

shaheeramunir · ‎08-05-2010

We also faced this issue few times but it was always resolved from E1/PRI service provider end so do take them in loop if you are sure that your PGs are stable.

texasjet79 · ‎08-06-2010

What on the E1/pri was causing the issue. How did you resolve this?

texasjet79 · ‎08-06-2010

So it happened first thing yesterday morning on a few calls and then dissapeared the rest of the day. Agents went home for the evening, came back in this morning and a handful of calls had the same and similar symptoms and now they are gone. Maybe I just need to warm up the servers each morning first !!!! LOL

Here is another description:

1. When the issue happens, customer has noted that on-hold music is playing at the same time they can hear the agent.

2. Agent gets a fast busy.

3. Call requeues to another agent and works fine.

The other is multiple agents answer but only one call gets pegged up and works.

geoff · ‎08-06-2010

On your contact center lines, do you have the busy trigger 1 and no call waiting, no Unity?

Regards,

Geoff

Anthony Holloway · ‎08-06-2010

texasjet79 wrote:

Here is another description:

1. When the issue happens, customer has noted that on-hold music is playing at the same time they can hear the agent.

2. Agent gets a fast busy.

3. Call requeues to another agent and works fine.

The other is multiple agents answer but only one call gets pegged up and works.

This is word for word, what I have been told, but in UCCX. What's goin' on?

khambetea · ‎08-06-2010

If you are using extension mobility, I would check to see if the agent was logged in to multiple phones. I would route plan report in

the call manager.

matt.harte · ‎08-20-2010

Hi All, does anyone know the TAC case number?

Thanks

Anthony Holloway · ‎08-27-2010

Yeah, where is the resolution for this issue? someone post back if they have had a successful fix.

texasjet79 · ‎08-31-2010

So looking at packet captures we noticed some bizarre behavior related to ARP on

the CVP Servers. ARP was periodically refreshing for no reason and it caused partial packet loss. The fix was adding to Registry entries and that resolved it. Apparently this was fixed in 8.x, unfortunately it took TAC two weeks to escalate to development and once they did there was an immediate fix provided.

Hope this helps.

Here is a write up from cisco that resolved OUR issue:

Two registry values are affected:

ArpCacheLife

ArpCacheMinReferencedLife

1. On the CVP server, in regedit, change the value of ArpCacheMinReferencedLife/ArpCacheLife

Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters to the maximum (0xFFFFFFFF)

2. Bring the CVP services out-of-service.

3. Reboot the CVP machine.

4. Bring the CVP services back into service.

If the parameter is not there by default, add it manually:

* Right click in the window and add the value (ArpCacheMinReferencedLife, ArpChacheLife) as a DWORD value. Then set it to 0xFFFFFFFF by double clicking on it.

Microsoft acknowledges ARP implementation "problems" in the Win2000 Server and WS2003 TCP/IP stack.

ARP functionality in WS2008 has been completely rewritten and this issue has been resolved in that rewrite.

Once this registry item is set to 0xFFFFFFFF, the problem goes away completely.

This problem has been in the Windows TCP/IP stack since Windows 2000 Server.

There is no patch for this from Microsoft as they will not issue any new patches for WS2003.

Setting this registry value is the only workaround and doing so should not have an adverse impact on normal operation.

This can also occur on ICM Servers apparently.

The crux of the problem is that when the ARP cache is being refreshed, ALL outbound messages are held in the transmit queue until it is complete. Once the cache is refreshed, normal message flow resumes. At the TCP level, you'll see duplicate ACKs and retransmissions (in a sniffer capture) likely because of (TCP) timeouts. At the app level, we see MDS round-trip times go through the roof, queues back up and significant latency with high-priority messaging. It is particularly problematic on PG private nets because of the volume of messaging between PGA/B.