Unity Connection (CUC) Cross Cluster Login Issue

Unanswered Question
Mar 18th, 2010

Greetings,

I am testing several scenarios in my lab with two separate (non-clusted) CUC nodes and I have come across some peculiar behavior with cross-cluster login and cross cluster transfer.

CUC Version: 7.1(3b)SU2  (7.1.3ES26.32900-26)

Configuration:

CUCM 7.1(3b)SU2  (two node cluster)

CUC-A  (single node CUC)

CUC-B  (single node CUC)

For sake of this discussion, assume the following:

PhoneA - Cisco IP phone with extension configured on CUC-A

PhoneB - Cisco IP phone with extension configured on CUC-B

PhoneC - PSTN phone (or Cisco IP phone with no voicemail)

PhoneD - PSTN phone with DN set as an alternate extension for PhoneA on CUC-A

What works:

1. Transfers from CUC-B to CUC-A (and vice-versa) works.  Example, PhoneC calls CUC-B and gets general greeting.  PhoneC enters extension for PhoneA and is transferred.

2. Users on CUC-B sending messages to CUC-A (and vice-versa) works.  Example, PhoneA calls CUC-A and logs in.  PhoneA user chooses to create a message (opt 2) and adds a recipient from CUC-B (e.g. PhoneB user).  Message sent and received OK.

3. Users calling from phones that are not configured as a subscriber extension or alternate extension works.  Example, PhoneC calls CUC-A and presses "*" to enter Sign-in dialog, user provides PhoneB extension and PhoneB voicemail password.  CUC-A says "one moment please" and successfully sends caller to CUC-B.  User is logged in and hears the review messages prompt.

4. Callers with DNs that are configured as alternate extension can access their voicemail box if they call the correct CUC server.  Example, PhoneD calls CUC-A.  CUC-A prompts the caller for this voicemail password (i.e. CUC-A does not ask for mailbox extension).

What doesn't work:

1. If PhoneA calls the pilot number for CUC-B, the CUC-B server says "one moment please".  At this point, CUC-B is trying to log the user into CUC-A but this fails.  The issue happens if PhoneB calls CUC-A as well (more details below).

2. If PhoneA forwards their phone to CUC-B and any caller calls PhoneA, they land on CUC-B and receive OpenTrees (standard opening greeting).

3. If PhoneD (which is an alternate extension for PhoneA on CUC-A) calls CUC-B, the CUC-B server says "one moment please". At this point CUC-B is trying to log the user into CUC-A but this fails. (more details below).

More Details:

I used "Remote Port Status Monitor" on both CUC-A and CUC-B during testing.  From these tools I can see that in failure scenario #1 and #3 (under "what doesn't work") are doing the same thing.   Using the examples provided, CUC-B recognized the extension (primary or alternate) correctly.  It then attempts to connect to the CUC-A server and execute a login sequence.  The CUC-A server asks for the password but the CUC-B server never asked the user for this information, therefore it can't provide the password (maybe it sends blank?).  In either case, the login fails and the callers receive the message "sorry your cisco unity connection home server cannot be reached at this time".  After this, they receive the open trees greeting for CUC-B.

Interestingly enough, if you do either #1 or #3 more times than allowed by your user's credential policy, you can lockout the user account.  That's awesome.

For issue #2, the CUC server receiving the initial call setup receives the proper CLID information but doesn't appear to be checking extensions that are in partitions from the other CUC system.  I have checked partitions and search scopes and I don't see a conflict.

The Use Case and Why is this a problem?

The use case illustrated by issue #3 under "what doesn't work":  Customer has two CUC clusters for 15,000 users.  They have a toll free number they give to users to call into voicemail.  This toll free number points to CUC-A but there are users on CUC-B that will call that toll free number.  If we stop here, then there is no issue.  BUT, if the user wants to add their mobile phone or home phone as an alternate extension, then no CUC-B user will be able to use the toll free number to access voicemail.  Worse, if they try it a few times (which they likely would) then their account will get locked out.  While I am sure that this isn't a universal use case, I definitely don't think it is unique.

The use case for issue #1 is similar to #3 but it is less likely to be a problem.  Internal phones will likely hit their messages button which is mapped to the correct server.  Communications can be provided to ensure people know their correct internal number (if needed) and, in the worst case scenario, a translation can be used to "trick" users into dialing the correct number. 

While issue #2 by itself is not a viable use case, what it tests actually may be.  The scenario is "overflow".  Let's say the customer uses IMAP.  This limits them to 72 ports on the mcs7845.  Further assume the customer needs more than 72 ports for audio during peak busy hour and/or burst.  Now, if the cluster is healthy we can overflow to the second server in the cluster.  But if the cluster takes a hit and the customer wants to "overflow" to the second cluster, forwarded calls will not integrate. I assume that one could try to use call handlers to pick out certain NPANXX ranges and then have the call handler transfer to a CTI RP on the CUCM using some funky prefix which will send the call to the other cluster.  But I haven't tested that yet and it is kinda ugly.

OK, so this was long winded but I don't like thread-banter so I figured I would provide all details.  While I could see where issue #2 and my argued use case is probably outside of scope for the feature provided by CUC cross cluster transfer/login.  I definitely think the use case for issue #3 is valid and I don't see anything in the documentation that suggests it wouldn't be.  So, in this case I am either running into a defect or I have a configuration issue.  Therein lies my question.

Thanks for tuning in and thanks in advance for any advice you provide.

Regards,
Bill

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Markus Schneider Fri, 03/19/2010 - 08:38

Thanks for all the details in your problem description!  Did you set up all the parameters defined here: http://www.cisco.com/en/US/docs/voice_ip_comm/connection/7x/administration/guide/7xcucsag247.html#wp1094475 on BOTH?

In particular, look at the "Respond to Cross-Server Handoff Requests" checkbox (along with the other cross-server handoff stuff that has to match under Advanced > Conversations), and that the pilot number for the corresponding server is both correct and that based on the CSS for the Connection server port/trunk in CUCM that it can reach the Partition/pilot number of the remote server.

For the failed cases #1 & 3, when you dial the pilot number, it should recognize that the caller is doing a sign-in (based on the calling number & corresponding CSS/Partition in CUC); then it should transfer the caller ("one moment please") to the other server, where the subscriber is homed to and there ask for the PIN (assuming it reached the Sign-in conversation based on its CSS/Partition for the inbound call).  The call, however, has been completely transfered to the subscriber's home server before the PIN is requested.  The only thing that the original called CUC node does it tell the home server what the calling number is, so it doesn't have to ask for that as well.

All cross-server transfer does is recognize that the caller is a subscriber on a remote system (using the CSS/Partitions), then transfer the call and send some digits (B & D by default) back & forth to followed by a string of digits that includes the calling number.  The receiving end takes that, matches it to a local subscriber's phone number, and then asks for the PIN (after the transfer has been completed.  The rest is just making sure the call from one Connection server to the other is routed properly and that the CSS/Partitions both in CUC and in CUCM are set properly.

So I'm a bit confused.  You said that it can actually lock the account.  That seems to imply that it's asking for the PIN; which means the transfered has been completed to the home server.  Or do you have the setting so that it automatically logs the user in with no PIN?

If the transfer fails (i.e. before the PIN has been requested), it should tell you "sorry your cisco unity connection home server cannot be reached at this time", which it sounds like you were hearing (and indicates a transfer failure).  So I'm not 100% sure what's going on there.

You could have something really strange going on, though, if--for example--you had the same DN in both servers but in different partitions.  Then the CSS would determine whether a cross-box transfer would be initiated or a local authentication.  I don't think something like that is going on, but just thought I'd point it out since you could really have some squirrly things happening if that situation exists.

If all this fails, the base Connection traces should be fine.  You should be able to see look at the latest CuCsMgr trace files (if you log into the CLI and do a 'file list activelog cuc/diag_CuCsMgr_0* detail') and if this transfer is happening. 

As for failure case #2, I think that's just working as designed; or at least it's not a call flow that cross-box login/transfer is supposed to address.  In the use case, if you were to overflow (due to excessive traffic) to a second cluster, it wouldn't be the best idea for that side to try to transfer the call right back to the (overburdened cluster).  It might just make the situation worse--for both clusters.

William Bell Fri, 03/19/2010 - 11:28

Markus,

I have checked the configs as provided in the sys admin guide.

For failed cases #1 and #3, what I see is that the first CUC server (CUC-B) detects the calling number (as you describe) and then initiates the cross-server sign-in.  I then see that the target CUC server (CUC-A) accepts the cross-server exchange (the B and D) and then requests the PIN.  Like so (from CUC-A):

01:45:43, -->SubAuthenticate
01:45:43,         State - SubAuthenticate.cde!TryCounter
01:45:43,         Event is [NULL]
01:45:43,         State - SubAuthenticate.cde!GatherID
01:45:43,         Event is [FalseEvent]
01:45:43,         State - SubAuthenticate.cde!LoadSubscriberMinimalData
01:45:44,         Event is [NULL]
01:45:44,         State - SubAuthenticate.cde!GatherPIN
01:45:44,         -->SubAuthenticatePW
01:45:44,                 State - SubAuthenticatePW.cde!ValidatePwd
01:45:47,                 Subscriber sign-in failed. Alias - fflintstone. Extension - 4161343807. Caller Id - 5551213.
01:45:48,                 Event is [FalseEvent]
01:45:48,         <--SubAuthenticatePW
01:45:48,         Event is [FalseEvent]
01:45:48,         State - SubAuthenticate.cde!TryCounter
01:45:48,         Event is [NULL]
01:45:48,         State - SubAuthenticate.cde!GatherID
01:45:49,         Event is [HangupEvent]
01:45:49, <--SubAuthenticate

The SubAuthenticate.cde!GatherPIN does NOT play out a prompt in my test scenario.  During this exchange my caller is simply on hold.  The corresponding trace from CUC-B for the same call (a snippet):

01:45:39, State - SubSignIn.cde!AuthenticateUser
01:45:39, -->SubAuthenticate
01:45:39,         State - SubAuthenticate.cde!TryCounter
01:45:39,         Event is [NULL]
01:45:39,         State - SubAuthenticate.cde!GatherID
01:45:39,         Event is [CrossBoxHandOff]
01:45:39,         State - SubAuthenticate.cde!RunHandOffRequest
01:45:39,         -->HandOffRequest
01:45:39,                 State - HandOffRequest.cde!SetupHandOffRequest
01:45:39,                 Event is [TrueEvent]
01:45:39,                 State - HandOffRequest.cde!IsWaitPromptEnabled
01:45:39,                 Event is [TrueEvent]
01:45:39,                 State - HandOffRequest.cde!PlayPleaseWait
01:45:40,                 Event is [NULL]
01:45:40,                 State - HandOffRequest.cde!RequestHandOff
01:45:43,                 Hand off request,  sent [B]
01:45:48,                 Hand off response,  waiting for [D],  received []
01:45:49,                 Event is [NULL]
01:45:49,                 State - HandOffRequest.cde!LogFailureOnRequest
01:45:49,                 Event is [NULL]
01:45:49,                 State - HandOffRequest.cde!HandleFailureOnRequest
01:45:49,                 Event is [LogonFailure]
01:45:49,                State - HandOffRequest.cde!PlayLogonFailure
01:45:53,                 Event is [NULL]
01:45:53,         <--HandOffRequest

So, you can see the hand off initiated by CUC-B to CUC-A.  The "B" and "D" are sent.  Then you see a LogFailureOnRequest and the resulting Handle Falure.  The PlayLogonFailure comes through to the caller as "your home server is not available" (paraphrase).  Then the caller is sent to next conversation, like so (trace on CUC-B):


01:45:53,         Event is [NULL]
01:45:54,         State - SubAuthenticate.cde!RouteToNextConversation
01:45:54,         Event is [NULL]
01:45:54, <--SubAuthenticate
01:45:54, Event is [NULL]
01:45:54, AttemptSignIn
01:45:54, State - AttemptSignIn.cde!Dummy
01:45:54, Event is [NULL]
01:45:54, PHTransfer
01:45:54, State - PHTransfer.cde!LoadInfo
01:45:54, Event is [TrueEvent]
01:45:54, PHGreeting
01:45:54, State - PHGreeting.cde!PlayGreeting
01:45:54, Call answered if needed
01:45:54, Playing greeting for Call Handler:  Opening Greeting

Now, this is very different from a scenario where the PSTN caller's DN does not exist as an alternate extension.  In that scenario, CUC-B answers the call and (of course) does not identify the caller as an existing subscriber.  We get an Opening Greeting.  We hit "*" to go to sign-in and then we provide an extension.  Then we see the following on CUC-B:

02:21:11, -->SubAuthenticate
02:21:11,         State - SubAuthenticate.cde!TryCounter
02:21:11,         Event is [NULL]
02:21:11,         State - SubAuthenticate.cde!GatherID
02:21:19,         Event is [CrossBoxHandOff]
02:21:20,         State - SubAuthenticate.cde!RunHandOffRequest
02:21:20,         -->HandOffRequest
02:21:20,                 State - HandOffRequest.cde!SetupHandOffRequest
02:21:20,                 Event is [TrueEvent]
02:21:20,                 State - HandOffRequest.cde!IsWaitPromptEnabled
02:21:20,                 Event is [TrueEvent]
02:21:20,                 State - HandOffRequest.cde!PlayPleaseWait
02:21:21,                 Event is [NULL]
02:21:21,                 State - HandOffRequest.cde!RequestHandOff
02:21:24,                 Hand off request,  sent [B]
02:21:27,                 Hand off response,  waiting for [D],  received [D]
02:21:29,                 Packet sent [043807#]
02:21:29,                 Event is [TrueEvent]
02:21:29,                 State - HandOffRequest.cde!DoHangUp
02:21:29,                 Event is [HangupEvent]
02:21:29,         <--HandOffRequest

At this point in time CUC-B, CUC-A appears to be checking the extension and then sends the caller to sign-in.  At this point CUC-B is out of the call.

02:21:24, Call answered if needed
02:21:24, Playing greeting for Call Handler:  Opening Greeting
02:21:24, DTMF received [B]
02:21:26, Hand off request,  received [B]
02:21:26, Event is [NULL]
02:21:26, HandOffResponse
02:21:26, State - HandOffResponse.cde!RespondToHandOffRequest
02:21:26, Hand off response,  sent [D]
02:21:29, Packet received [043807#]
02:21:29, Event is [TrueEvent]
02:21:29, State - HandOffResponse.cde!ProcessPacket
02:21:29, Event is [RouteToLogon]

Then CUC-A collects the PIN:

02:21:29, State - SubSignIn.cde!AuthenticateUser
02:21:29, -->SubAuthenticate
02:21:29,         State - SubAuthenticate.cde!TryCounter
02:21:29,         Event is [NULL]
02:21:29,         State - SubAuthenticate.cde!GatherID
02:21:29,         Event is [FalseEvent]
02:21:29,         State - SubAuthenticate.cde!LoadSubscriberMinimalData
02:21:29,         Event is [NULL]
02:21:29,         State - SubAuthenticate.cde!GatherPIN
02:21:29,         -->SubAuthenticatePW
02:21:29,                 State - SubAuthenticatePW.cde!ValidatePwd
02:21:36,                 Subscriber sign-in successful. Alias - fflintstone. Extension - 4161343807. Caller Id - 5551213.
02:21:36,                 Event is [TrueEvent]
02:21:36,                 State - SubAuthenticatePW.cde!ReturnAuthenticated
02:21:36,                 Event is [Authenticated]
02:21:36,         <--SubAuthenticatePW
02:21:36,         Event is [Authenticated]
02:21:36,         State - SubAuthenticate.cde!ReturnAuthenticated
02:21:36,         Event is [Authenticated]

The user is logged in and happy.

The only difference between the succesfull even and failure event is that in the failure event I have the 5551213 configured as an alternate extension on the subscriber's mailbox.  My observation is that CUC-B detects this correctly and hands off to CUC-A, but for some reason it does not transfer the call to CUC-A to complete the login process.  BUT, from CUC-A's perspective it is asking for a PIN and is either not receiving one or CUC-B is sending something bogus (I presume the former).

Now, you asked about account lockout.  The login attempt in the failure scenario counts as a failed login attempt.  So, if you have your credential policy set to lock the user account after three failed login attempts and the user tries to call the pilot three times, the account will get locked.  At no time has the user been given the opportunity to enter the "wrong" PIN/password.

I pulled the CUCsmgr traces like you said.  They basically confirm the same thing.  I have them if you want to see them.

Thanks,
Bill

Markus Schneider Fri, 03/19/2010 - 11:59

But look, in the failed case:

01:45:43,                 Hand off request,  sent [B]
01:45:48,                 Hand off response,  waiting for [D],  received []

whereas in the working case:

02:21:24,                 Hand off request,  sent [B]
02:21:27,                 Hand off response,  waiting for [D],  received [D]

Seems like for whatever reason that digit either didn't get sent (maybe it was in a different call handler than Opening Greeting)?  It looks like on the transfer, the calling party 5551213 sent the call to the fflintstone's sign-in conversation, instead of Opening Greeting:

01:45:47,                 Subscriber sign-in failed. Alias - fflintstone. Extension - 4161343807. Caller Id - 5551213.

Since normally the originating side sends B, then receives D, upon which it sends the extension#, e.g:

02:21:29,                 Packet sent [043807#]

Is this integrated to CUCM via SIP or SCCP?  Any chance you have 55551213 configured as one of the voice ports or something?  Otherwise, take a look a little earlier in the rPSM trace to see what the calling number is on the Connection server receiving the cross-box transfer.

Actions

This Discussion