cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1751
Views
5
Helpful
26
Replies

IPCC - Design to Support DR

dtran
Level 6
Level 6

Hello all,

Has anyone deploy IPCC enterprise to support DR ? if so, please share your exprience on your deployment !!!

I am having a hard time getting my IPCC environment to work as design. I have a duplexed IPCC 6.0 enterprise environment with side A and side B located at separate locations connecting via a full DS3 MPLS connection and I am having a hard time getting the IVRs to go active when PG1a is offline. When I shutdown PG1a the IVR PIMs on PG1b do not go active for some reason and I get a busy signal on all of my tollfree numbers.

Please share your experience if you have ran into an environment like mine !!! Thanks very much in advance !!!

Danny

26 Replies 26

cherilynn1030
Level 1
Level 1

We are running IPCC enterprise 7.1.5. duplexed system pg's, dual IVR's. Side A is located at our primary, and side b is at our DR location. we have a DWDM connection, so straight fiber to our DR location.

do you have a jtapi connection for side b configured in call manager? When side a is active, what do your process windows reflect (in regards to side b.)

Hi Cheri,

I have split CCM cluster so I do have 2 CCM Subs at the DR site and PG1b and IVR2 are talking to the CCM Sub at the DR site. Side A is always active and side B shows idle as far as the processes is concerned. I tested other components and they worked when side A is offline. When I shutdown RoggerA, RoggerB goes active and everything still work and the same with IVR1 when IVR1 goes offline IVR2 goes active.

For some reason the IVRs couldn't go active when PG1a is offline.

My next question is: In a duplexed environment when you run setup on the PG. Do you have to select the preferred side (A or B) or you select No side preference ?

Thank you very much !! I appreciate your response !!!

Danny

Danny,

Because you are running ICM v6.0, you have some restrictions in how the system is designed. I suspect that you might have a mis-configured system.

I'm assuming this design:

Site 1: RoggerA, PG1A, IVR1, CCM Sub1, CCM Sub2

Site2: RoggerB, PG1B, IVR2, CCM Sub3

How many PIMs do you have configured on PG1A and on PG1B and what devices are they connected to?

/Jeff

Hello Jeff,

Below is exactly what I have

Site 1: RoggerA, PG1A, IVR1, CCM Pub, CCM Sub1.

Site2: RoggerB, PG1B, IVR2, CCM Sub2, CCM Sub3.

I agree with you that there might be a mis-configured setting or settings somewhere. I re-ran setup on the Roggers and the PGs but couldn't find anything.

I have 3 PIMs on each PG (CCM PIM, IVR1 PIM and IVR2 PIM). My setup is pretty straight forward but don't know what I have done wrong here. I am open to any suggestions !!

thank you very much !!! I appreciate the response !!!

Danny

Hi Danny,

you need an additional, dummy PG, to achieve full redundancy, no need to have a PIM active on that one, establish a side preference, this would allow the CC to stay up and connected to a majority of PGs if one side goes out of service(basically your additional PG will connect to the active CC side, being A or B and allow for it to be working, this is fully described in the SRND).

Second point, I hope the MPLS is just used for the connection between CC and PG, if it is used also for private links then your network configuration would not be supported, since it would not be able to deal with the ICM high priority traffic latency and failover requirements.

Third, what is going exactly out of service in the failover scenario? PG? PIMs? How is the PG seen by the CC? Is PGAgent still in service?

Regards,

Riccardo

PS Remember to rate useful posts accordingly please.

Ok, so the basics first, we want to make sure that the PGs are correctly running in Duplex. There are lots of ways to determine this but we'll just look at a couple of simple indicators. We will look at the 2 of the processes on each of the PGs.

- On the PGs, look at the title bar for the MDS (Message Delivery Service) process. One PG should show “InSvc PR-Enb Clk” and the other should show “InSvc Pr-Dsb Clk”. We're looking for MDS to be in service, they are Pr (paired) while one MDS is enabled while the other is disabled. A normal functioning system (with event tracing at 0 - or normal) won't have many scrolling events in the process window.

- On the PGs, look at the title bar for the PGAG process. One PG should show “InSvc A:Active B: Idle” or “InSvc A:Idle B: Active” while the other PG will show “Not Active”. This shows that one of the PGs is InSvc or actively connected to the Call Router and which call router it's actively connected to (A or B). The other PG should be Not Active. The non-active side (if you were to view the actual logs) would show that it has 3 idle connections to Call Router A and 3 to Call Router B. A normal functioning system (with event tracing at 0 - or normal) won't have many scrolling events in the process window.

- In both cases, your most recent log events in these windows should be showing something along the lines of 'reporting metering statistics'. If it looks like either of these processes is trying to do something with events like connection retry or connection failed then we've got a low layer problem.

Let's start there and let us know.

/Jeff

Hello Jeff,

I checked the processes on both PGs and all seem normal, pretty much exactly like what you described. I do not see any connection failed or connection retry messages but see a lot of "reporting meter statistics" messages. All seem normal when both sides are up.

Thank you very much !!! I really appreciate your response !!! Please let me know what else I should look for !!!

Danny

Ok Danny, that's good. Now we need to try to figure out why the PIMs won't go active. This part is service impacting.

Some background: PGs are a real-time synchronized design in that MDS is busy populating the OPC data between both sides of a duplex pair. You can actually watch messages occur in both OPC windows from the PIMs regardless on which side the PIMs are active on. It's also completely natural for the active PIM to move from one side to the other based upon network events, i.e. don't always expect the PIM to only ever be active on any one side. It is also completely natural for a PG with multiple PIMs to have the active PIMs shared between the two sides, i.e. not all on side A or not all on side B.

I would like to try to see if we can get each of the PIMs active on the B side but individually. Remote into both PGs and confirm that all the PIMs are active on PG1A. Here is the service impacting part but we're only doing it to one peripheral at a time. On both PGs, open the PIM3 window so that you can watch what's happening. PG1A is active and PG1B should show idle. Crash the PIM3 process on PG1A by clicking on the X on the window, closing it. PIM3 on PG1B should immediately try to connect to IVR2, and ideally go active. The PIM3 process on PG1A will be restarted by Node Manager after a few seconds and will be in an idle state. If PG1B PIM3 doesn't go active it will eventually switch back to PG1A or you can crash the PG1B PIM3 process and it will go over to PG1A post haste.

If PIM3 didn't go active on PG1B, then dump the logfile and post it up.

Open a command prompt on PG1B and type:

C:\cdlog pg1b

Such as C:\ cdlog cust pg1b

This will bring you to the ICM logfiles directory (you can also cd down to it \icm\cust\pg1b\logfiles)

Now type:

dumplog pim3 /last /o

this will dump the logfile into a txt file named pim3.txt in this directory. Post the logfile up here and let us know what time you did the failover test.

Feel free to do the same for PIM1 and PIM2 to try to get them to go active on PG1B. I'm just trying to focus each problem at a time in baby steps. Riccardo has valid points but that's a bit higher in the stack.

/Jeff

P.S. what is your private network between the PGs?

Danny,

You probably want to try sub2 to site1 or like Ricardo stated add 3rd PG this will keep the connectivity.

Esentially you have all components are working independently but not in sync.

The third PG can be any where.

Good luck.

Hello Jeff,

Sorry, for not getting back to you sooner I was out of town for a few days.

I am planning on scheduling an outage at the end of this month. I will test out your scenario and gather logs.

The private network and the visible network are connecting to the same WAN link (a full DS3 MPLS circuit) with QoS enabled.

I understand it's difficult to say anything without the logs but do you have any ideas what might be causing the IVR PIMs not go active on PG1b ?

Thank you very much !!! I really appreciate your responses !!!

Danny

Well Danny,

your design is not supported, sorry to tell, did you get a specific BU agreement?

You could verify it in the SRND.

Otherwise I would suggest you to check with your local sales rep and who deployed it.

Regards,

Riccardo

rbua: is the design not supported based on his version of ICM 6.0 ?

It is not supported to share the same WAN link for Private and Public connection, further more if it is a MPLS one.

Recently we made some exceptions at the A2Q process in the event latency and redundancy could meet some specifics, which are specified in the 7.x SRND.

Regards,

Riccardo

Danny,

There is an opportunity for there to be a few things wrong. The basics would be PIM setup and make sure that the correct IP Address is in there for the IVR and that you can ping the IVR from the PG. Is the TCP port correct? Each IVR only has one TCP port for the ICM Sub-System, typically port 5000. Is the PG setup for Service Control Interface? Is the Peripheral ID in the PIM correct (based on your design, I would guess 5002 for PIM3)?

My intent was to take some very basic troubleshooting steps. While I agree that this design, as described, doesn't comply with SRND standards, there is no reason that PG1B shouldn't be connecting to the PIMs. We can work on systemic failover next as well as articulate design scenarios around dummy PGs and network link separation based upon your needs.

Where are your A and B sites? Same city, same state or cross country?

/Jeff

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: