I am having a reoccurring issue (monthly) where one of our business applications goes down and causes high call volume into the contact center. There are 53 CTI Ports and they are all set to go to voicemail for forward busy/no answer, where there is a message "due to high call volume..." This was tested and worked fine in a small queue of 10 ports. However, on the larger queue, the high call volumes crashes the callmanager subscriber that the agents are on, and callers are getting a busy signal instead of going to voicemail and "high call volume" displayes on the agents phones.
In the ICM router log viewer it shows "No free routes to send call to translation routes.... all 10 routes are in use" in the router event viewer it shows "Translation route timeout". Meanwhile the ICM script monitor shows a spike in calls about 80 thousands calls in less than 20 minutes. It seems like there is a loop somewhere, because even if all our customers were to call at the same time it would not even reach 10,000 calls.
I have 10 Translation Routes for 10 CTI Route Points and 53 CTI Ports for this particular contact center. The CTI Route Points are set for Max Calls = 5000 and Busy Trigger = 4500. Should'nt this be lowered to much the number of available ports? or should the CTI Route Points be set to go to voicemail on busy aswell?
UCCE 7.2(6), IPIVR 4.0(5), CCM 4.0(2)
You certainly have a problem somewhere.
When the call hits the route point for the start of the trans route, it waits until the Router responds with a real target (the CTI port) and sends the data to that PG so it can tie them together, then sends the call. If you have 53 CTI ports, there is no reason not to have 53 route points as these are free and it's just a matter of creating the RPs, mapping them to JTAPI, and running the Trans Route Wizard to add the bigger pool of DNIS.
Since you have a bug somewhere, you will probably end up in the same situation but the message will say "No free routes to send call to translation routes.... all 53 routes are in use".
This is not going to solve your problem - just move it from saturating the route point pool to saturating the CTI port pool.
What is going on with those ports? Are they sending the call back into the system somehow?
Nothing on the ports, they do not go out of service... When calls spike, they all fail from the Translation Route to VRU node in the ICM routing script. Somehow, calls are being sent back to the systems. So it re-dialing in attempt to when connect to a target
To troubleshoot you can look at enabled scripts in ICM script editor and see if certain script(s) are carrying high call volume. Then look at the script to see if the call routing is a problem.
I have several routing scripts that use the same Network Trunk Group, but only the two main routing scripts for the call centers that support that business application are affected with high call volume. This happens once a month for a about 20 minutes... Others scripts on the on different Network Trunk Groups are never affected.
I'm assuming in your 2nd script if the translation route node fails you shoud release the call or send it to any available agent. Also, it would have been nice to post your scripts in monitor mode, with the node number next to them. Als, the 80000 calls you're seeing aren't calls, but passes through a specific node, it coule be one call which just passes over a loop 80000 or two calls which each pass 40000, etc.
Maybe we need to take a step back, I re-read your original message and it sounds like there's all sorts of things wrong with your environment. So, why don't you break up your issues into smaller logical pieces and take one at a time? For example, you mentioned that your UCM was crashing, that is one problem. You seem to be running out of trans routes, that's another problem. There are some infinite loops in your ICM script(s), that's another problem. While they all might be related you need to break it up in order to be more effective.