6500 switch acting as a hub

Unanswered Question
Jan 7th, 2009

We have 6500 switch running IOS s72033-adventerprisek9_wan-mz.122-18.SXF3.bin

Many servers are connected to this switch. When i use wireshark to sniff traffic on our

windows DFS server connected to this switch, i see traffic for entire VLAN on wireshark. I

am not suppose to see traffic between other hosts/server since this server port is not

configured as a span port. It seems like 6500 switch is acting as a hub . There is no span

session configured on this switch. Our system admins have been complaining about

performance issue on their servers. I used wireshark on different servers on that switch

and i see same result. All the servers connected to this switch is seeing traffic for

entire vlan regardless of ports. I even connected my laptop to this switch and i can see

traffic for entire vlan.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
xcz504d1114 Wed, 01/07/2009 - 14:10

Switches will flood packets in which they don't know the destination to, out every port except their originating port, so you will always see some unicast traffic. Every 5 minutes by default the mac-address entries will age out, if it is excessive and you are sure you are actually seeing what you're claiming, check the mac-address tables to make sure there are entries being populated (show mac-address-table) and you can even adjust the default mac-address aging timers to a higher value.

Is there just 1 6509? Is there any HSRP? How many packets per second are you seeing on the port with wireshark? With wireshark, view the IO graph for a 10 minute capture to see if you are having huge packet bursts. Are the servers using dual NIC's? Do they have their NIC teaming setup properly (I see it all the time in my datacenter, inproper NIC teaming can cause mac-address issues, check your logs to see if you have mac flapping, that is a dead give away)?

Is there any multicast / broadcast traffic or is it all unicast?

Craig

altaf007 Wed, 01/07/2009 - 14:19

Hi Craig,

I see ip traffic, broadcast, eigrp update, etc between different hosts/server. I see their web, e-mail, http,ssh, telnet, etc.

Is there just 1 6509? We have two 6509 connected with layer two etherchannel.They both have same VLAN/subnet for servers

Is there any HSRP? yes,I have different vlan for HSRP. This vlan has different subnet.

How many packets per second are you seeing on the port with wireshark? I am seeing 20 to 50 mbps packets

Are the servers using dual NIC's? No dual nic. just single 1 gig NIC

Do they have their NIC teaming setup properly (I see it all the time in my datacenter, inproper NIC teaming can cause mac-address issues, check your logs to see if you have mac flapping, that is a dead give away)? I already checked that and i did not see anything in logs.

I

xcz504d1114 Wed, 01/07/2009 - 14:31

Kwu gave you a link to exactly what I was thinking about with the HSRP configurations (Asymmetric routing). One of the characteristics is large packet bursts at random intervals. There are 2 ways to over come this, one as recommended in documentation is to adjust the mac-address agining timer from 5 minutes (300 seconds) to 4 hours (14400 seconds) to match the ARP aging. Another method is to not manually load balance between your HSRP gateways, have one router be the default gateway for everything with the second router being the standby for everything.

Every port you will see broadcasts, EIGRP, and occaisionally you will see the other stuff. How often you see the other stuff is the concern.

To clarify, you are seeing 20 to 50 mbps? Is that Megabits per second? Million packets per second? I'm looking more for a count of how many packets per second you are seeing, not necessarily the bandwidth usage. But even 50 Mbps (Megabits per second) on a 1 Gigabit connection is insignificant. Now some NIC's handle "junk" traffic, I consider "junk" anything that is not relevant to that device that is recieving it, that is why I am trying to get an idea of how many packets per second you are seeing.

Craig

Yudong Wu Wed, 01/07/2009 - 14:15

Not sure if the issue is related to unicast flooding.

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801d0808.shtml

Also please verify with Server guys about:

1. how many actived NIC card on each server and how they are connected to switch.

2. If server has multiple NIC card, does it use one to send/receive traffic. You might aslo sniffer the incoming traffic on the port which the server is connected to to find out this.

altaf007 Wed, 01/07/2009 - 14:50

There is only one NIC 1gig on each server. I tried to configure 6500 switch with following command but it does not work

unicast flood protection. I am gonna open case with cisco TAC about this.

Giuseppe Larosa Wed, 01/07/2009 - 15:14

Hello Altaf,

it looks like the CAM table is full.

check with

sh mac-address-table count

the max is in the order of 65,000 MAC addresses

Last summer we had a strange problem caused by CSM that was creating random MAC addresses when the traffic volume was high (more then 2 Gbps) and high number of connections.

We solved this with a CSM firmware upgrade.

Are you using CSM and firmware is 2.3.x or similar ?

We now have 4.2(9) on them.

We opened a service request and they found the problem hitting a known bug very quickly.

Hope to help

Giuseppe

Actions

This Discussion