Ace 4710 strange behaviour

[cineca]

Hi, We have two ACE-4710-K9 (named LB01 and LB02) configured in HA mode. Besides Admin, on each of them there are tree context configured, named, ACADEMIC, COMMERCIAL, STREAMING. On LB01 the active context is ACADEMIC. On LB02 the active contexts are COMMERCIAL and STREAMING. Each context is configured with a FrontEnd and a BackEnd Vlan interface, and a "management" Vlan interface used for accessing and monitoring the device and for the downloading of the needed ssl certificates. Recently we upgraded the devices to Version A3(2.6) form a previous A3(2.4). After that upgrade we experienced some strange behaviour. From the context in STANDBY state we are not able to ping the host on the "management" Vlan interface, while there is no problem on the other Vlans. We see that the ICMP packets are sent to the Vlan, are replayed by the remote host BUT are not received at all on the LB01 or LB02. No messages in the log. Trying with 5 consecutive (failed) ping we can see that the counters of unicast packet output on LB01/LB02 Vlan is incremented by 5 BUT the unicast packets input counters is unchanged even if the remote host sent the replays. In the STREAMING context this behaviour isn't constant, ie the ping *sometimes* starts working for a few second and then returns to stop. In the other standby context the ping never works instead. In the active context all works fine. This strange problem prevents us to load the ssl certificates in the STANDBY context from the "management" Vlan. We was not able to find any reference to a similar problem in the Cisco documentation or Tac collection, so we are curious to know wheter someone else experienced such a behaviour. Thank you and best regards. Alessandro Asson - CINECA

[ddastoli]

Have you tried to switchover the context and check if always the standby one is facing this problem?

Have you tried to collect a sniffer trace from the TenGig of the standby ACE on the Cat6k to check if the packets are coming back from the servers in the management VLAN?

If you create a interface management VLAN in the Cat6k hosting the ACE, are you able to ping the standby ACE management IP?

[cineca]

Hi, Domenico

may be you are italian ?? :-)

Thank for your answer!

>Have you tried to switchover the context and check if always the standby one is facing this problem?

Not yet: I will try in the next days. I will let you know the outcome.

>Have you tried to collect a sniffer trace from the TenGig of the standby ACE on the Cat6k to check if the packets are coming back from the servers in the management VLAN?

>If you create a interface management VLAN in the Cat6k hosting the ACE, are you able to ping the standby ACE management IP?

We have the ACE 4710 appliance, not the Cat6k module..

Anyway, from the standby ACE appliance I'm able to ping its own management IP, but I'm not able to ping the management IP of the

active appliance.

I didn't try a sniffer trace collection, but pinging from the ACE a third router on the management Vlan with "debug imcp" enabled

I see the ICMP request recived by the router AND the ICMP replies sent back to ACE by the router. So I guess that such

ICMP replies are lost by the stanby appliance interface.

Thank You and Best Regards

Alessandro Asson - CINECA

[cineca]

>Hi, Domenico

>may be you are italian ?? :-)

>Thank for your answer!

>

>>Have you tried to switchover the context and check if always the standby one is facing this problem?

>

>Not yet: I will try in the next days. I will let you know the outcome.

I finally managed to switchover the context and I verified that the problem is always on the standby side,

ie: the problem followed the standby appliance.

Alessandro Asson

[ddastoli]

ciao Alessandro,

Can you send show run from the active and standby ACE (affected context by the problem), together with a network topology diagram?

Can you also attach a traceroute from the 'faulty' ACE to the rotuer/other ACE and back (from router/other ACE to the 'faulty' ACE)?
Also if you have captures/debug session taken on ACE/router please send it to me.

By the way, did you change anything in the config from A324 and A326?

If not, are you sure it was working with A324?

By the way, I think at least a workaround to load the SSL certificate to the standby would be to switchover and make the one active. Does this work?

Cheers,

/dom

[cineca]

Ciao Domenico,

I send in attach 4 files containing show running-config for LB1 and LB2, the network topology diagram and some test (ping) output.

I saw that for the standby context there is an arp problem (mac address not resoved by ACE/stby and on the routers d01/d02...

same MAC address for both IP aqddress of LB01 and LB02..)

[ddastoli]

Thanks,

I see you are using shared VLAN config in both ACE.

Same VLAN 1000 is used for both Admin and streaming context.

In this config, you may need to use the shared-vlan-host-id command as explained here:

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA3_1_0/configuration/routing_bridging/guide/vlansif.html#wp1025243

In fact as explained:

'By default, the bank of MAC addresses that the ACE uses is randomly selected at boot time. However, if you configure two ACE appliances in the same Layer 2 network and they are using shared VLANs, the ACEs may select the same address bank, which results in the use of the same MAC addresses. To avoid this conflict, you must configure the bank that the ACEs will use.'

This would also reply to your question in the readme file:

SHOW ARP TABLE ON THE D01,D02,D07 ROUTERS SHOWS THE SAME MAC ADDRESS FOR

BOTH IP ADDRESSES OF LB01 AND LB02: is that normal ??

Hope this helps,

Dom.

[cineca]

Hi Domenico,

thank you for your analysis: it seems to be the right solution..

I will try the new configuration as soon as possible.

Thank you and Best regards

Alessandro