Advice from switching experts

new_networker · ‎11-20-2008

Equipment & Topology

1. Perimeter ASA (In redundancy Active/Passive)

2. Dual Cat 6500 chassis (ACE-bridged, FWSM) with HSRP (all groups active on single chassis)

3. Vlans : 1, 2, 4, 5

Vlan 1 is outside MSFC. VLAN 2, 3, 4 are defined on FWSM which lies after MSFC.

4. Etherchannel is used between Cat 6500

5. The server hosts have dual NIC (Active/Passive) terminating on different Cat 6500s.

Problem Summary: Intermittent pings from all other networks to a 'single' host x only in vlan 2. All other networks are able to ping other hosts continuously in vlan 2. And ping to host x from other hosts in same vlan is also pinging successfully. So the issue is other networks.

While investigating I switched off the standby chassis and reloaded the primary chassis. After reload, the ping for host x in vlan 2 gave 'no' problems and successful throughout. After sometime, I switched on the standby chassis the pings started to break again. I couldn't find anything in the logs as well.

This setup has been in place for couple of months and started giving this problem only recently.

Could you please provide your valuable inputs and as many troubleshooting points as possible to isolate and resolve the issue.

Some of areas which I think should be investigated are

1. Switching loop

2. MAC address flapping

3. Another host with same IP connected to the switch.

Looking forward to your expert advice.

Jon Marshall · ‎11-20-2008

Don't claim to be an expert but if all the other hosts in vlan 2 do not exhibit the same behaviour as this particular host i think the configuration of the host is where you should be looking.

1) Switching loop - unlikely if only one host is affected

2) Mac-address flapping - worth a look.

3) Another host with same IP - definitely worth considering but again difficult to see why it would only happen when the standby FWSM comes up.

Other things to consider

1) Presumably when talking about a host in vlan 2 you are referring to a dual-honed server ?

Check teaming configuration on the server.

2) Check routing table on the server

3) Check the FWSM config to ensure that they are the same - should be if in active/standby mode.

4) if the setup has only recently begun to give problems haave any changes been made to the ACE/FWSM/6500 configuration. Have any changes been made to the server.

5) If you can reload the standby again check the arp table on the FWSM active and on server. When the standby comes back online check the arp cache of all three and see if there are any inconsistencies.

You say all HSRP groups are active on one switch. Is the same switch the STP root for all vlans ?

Jon

new_networker · ‎11-21-2008

Thanks.

All checks went thru fine. By the way, the problem is not seen any more even after bring up the second chassis. Scary if it happens in Prod.

Does it matter that the STP root is on the same switch which has HSRP active.

Is there any way to determine all the IPs currently connected to a VLAN.

new_networker · ‎11-21-2008

The only thing I remember that was changed during the problem was server NIC teaming. The teaming was broken and re-done.

The teaming in the servers is Active/Pass. Could it be the active standby issue. i had similar problem before i.e. port flapping with active/active etherchannel between server and switch. HOwever, I experienced port flapping log in the switch.

But for the given problem I didn't see any log message generation.

Also is there a way to check on switch whether packets are being sent via the passive NIC.

Jon Marshall · ‎11-21-2008

Apologies for delay in getting back.

As long as the relevant vlans are allowed across the trunk link connecting your 6500 switches then STP root position does not matter although it would make sense to have it match the HSRP settings.

You talk about active/active etherchannel from server to switch. Do you mean etherchannel or do you just mean 2 active conenctions. if you saw portflapping this could be due to the server using both NIC's to transmit data.

Can the server guys confirm that this server is running active/standy teaming configuration.

Jon

new_networker · ‎11-21-2008

I mean't two active connections and this was long time back. We had changed to active/passive to resolve port flapping issue.

The current config is active/standby on which the problem recently started.

Things we did to investigate

- Brought down chassis 2: No effect

- Removed all the cables from chassis 1 and only kept the problem host x and laptop: Still problem persisted.

- Changed the network cable connected to host x: Still problem persisted.

- Put back all the cables into chassis 1: Problem vanished.

- After sometime, brought the chassis 2 up: Problem started again

- Lastly, the teaming on server was broken and redone (Act/Pass). We checked the next day: Problem vanished and has still not re-appeared with the chassis 2 online.

Is there a way where I could check the cat 6500 switchport activity level. Basically I would like to cross-verify the configurations on server. In active/passive, I shouldn't be seeing any packet coming into the passive connection port.

Thanks