I am running UCCX Premium version 184.108.40.20601-24 in an HA environment. I have a 3 node CUCM cluster running version 220.127.116.1100-5.
We are using a Cisco 2951 as CUBE with 100 SIP Ports with automatic routing to an additional 3 PRI for a total of 169 channels.
Our customer is a power company using the UCCX for the Call Center and IVR Applications (Outage Reporting). We are doing a back end integration in to Oracle for gathering/reporting the power outage.
The issue that I am having is that when we have a power outage, we with in seconds are receiving 169 calls filling up the all PSTN trunks and staying full for a long time (greater than 1 hour)
The UCCX is answering all of the calls and processing the caller input.
We place the caller on hold(MoH), to do the database lookup (Complex lookup, can take 5 - 10 seconds), after the lookup, we are trying to unhold the call in the script and getting the following error message:
HELD; nested exception is: com.cisco.jtapi.PlatformExceptionImpl: Cti request timed out
This also happens when the script tries to transfer the call to the agent, causing the agents to go into reserved mode and having to close the CAD and log back in as the script has aborted the caller.
I have a case open with Cisco on the CUCM about the CTI Timeout, but I want to be able to deal with this in the script. The exception com.cisco.jtapi.PlatformExceptionImpl is not listed as exception that can be caught.
I want to be able to catch the exception, and delay and then retry to at sometime later, will the com.cisco.WFExecutionException be able to catch it? or since the exception is happening outside of the script, can anything be done to keep from dropping the caller?
Can anyone suggest a way to test high call volume?
Need help as the issue only shows up at the worst possible time.
can you please enable SS_TEL debug and post the logs to here?
I am afraid com.cisco.WFExecutionException is not a parent of com.cisco.jtapi.PlatformExceptionImpl; there might be a slight change something rethrowing the latter as an Exception recognised by UCCX but I would not count on it.
What you are writing about is actually very interesting and I will definitly test this in my lab. I believe what happens is when you place the call on hold you actually leave the CTI Port and when you unhold it there's a chance that there is no available CTI Port and the JTAPI request fails or times out.
We have over written our logs, so I can't provide the logs for the last time that we had this occur (09/29/2014). TAC transferred the logs via the WebEx and they are not attached to the case.
I did forget to mention that every time that we have this happen, we are getting a Code Yellow on the CUCM that is the main call processor for the UCCX.
TAC has indicated that the CUCM is unable to write all of the logs, cdrs, etc under the high call volume. I am following up behind another Cisco partner and I do not believe that the UCS were sized correctly for my customers call volume. They are running Cisco UCS C220-M3 LFF with the 7.2 SATA drives. We have virtually disabled all logging on the CUCM as to lower the impact of all of the holds, transfers, etc.
The previous partner had written a lot of custom JAVA for doing the database and backend API integration. This took a very long tome to execute on each of the calls, 20 -30 seconds or longer per call. I have rewritten scripts to do all of the integrations, via subflows and reduced this time to 5 - 10 seconds.
In the mean time, I have removed all instances of placing calls on hold, by playing prompts instead of placing them on hold, where I can. I have also combined scripts as to reduce transferring from script to another script. So I have lessened holds and transfer by about 60%. We have been unable to tell if this working or not, as we have to wait for the next large power outage.
The one place that I am unable to remove placing the caller on hold is when, we do the database dips or call the backend API(HTTP), any suggestions?
Any help, pointers, information would be greatly appreciated as the customer is getting very frustrated.
I was under the impression that when the CTI port placed the caller on hold, that it was still reserved for that contact to return from being on hold or transferring?
thanks for this detailed explanation. I can understand your struggle and I believe you have already heard this like a dozen times.
There's no good workaround to this, but here's what I would suggest: forget the Database Subsystem altogether. It's very powerful as we all know but it places unnecessary load on the UCCX in this situation. I would build a light HTTP-to-DB proxy somewhere real close to the UCCX server and use it to persist data asynchronously (if it's not absolutely necessary to tell the caller whether the DB operation was succesful or not). I would not even care to send a nice XML back to UCCX, just a plaintext document to see whether the HTTP "proxy" is alive or not.
The key is to kick the call through the system as quickly as possible.
I think the CTI port is still reserved but the JTAPI handle is freed up - again, this is just a wild guess. I will take a look at this but I will need to have some time to explore the internals.
We will look into the Proxy, the caller needs to be informed if the information they entered was correct or not, so we must have a return from the DB or the API.
So from a load perspective on the UCCX, the HTTP to Proxy would be less of a load than calling straight to DB? I have never done a comparison. I know that the DB calls are much faster than the HTTP API calls that I am doing.
By the way, I do appreciate all of your posts on this forum.
there's nothing wrong with the DB subsystem. There's nothing wrong with JDBC, and naturally, there's nothing wrong with pooled database connections. However, under the circumstances, watching a pool of connections, creating new ones and destroying abandoned/idle ones is something I would happily give up, even for the higher price of creating HTTP connections.
Some programmers argue that creating a HTTP thus TCP connection is slower than sending a simple message over an already established connection (that is, one of the connections of the JDBC pool). That's absolutely correct. However, it is still simpler and eats up less CPU cycles and requires less RAM (and it's most certainly easier on the Garbage Collector process) than the DB connection. Yes, I know, now I am supposed to give you the numbers and I wouldwill try to do the profiling as soon as I have the opportunity.
P.S.: I can help you with creating that HTTP "proxy", the Grails framework that I use is ideal for this task.
The short answer is that you don't.... That isn't entirely true while at
the same time it kind of is, but for the most part you don't configure
the softkeys. You enable or disable them via TCL. Here is the long
answer. Be sure to read the whole thing or e...
Topology: IP Phone > Switches > Microsoft NPS setup to forward 802.1x
proxy to > ISE 2.1 patch 3 Authentication: EAP-TLS using Cisco MIC SANs
Phone Models 802.1X support? 802.1x flavor Addtl Comment EAP-MD5 EAP-TLS
Cisco 3905 Y Y N Cisco 6911 Y Y N Cisco ...
This document describe how DST changes and how time changes are
implemented in DST. Daylight Saving Time (DST) is the practice of
setting the clocks forward 1 hour from standard time during the summer
months, and back again in the fall, in order to make b...