6500 switch acting as a hub

Unanswered Question
Jan 7th, 2009
User Badges:

We have 6500 switch running IOS s72033-adventerprisek9_wan-mz.122-18.SXF3.bin

Many servers are connected to this switch. When i use wireshark to sniff traffic on our

windows DFS server connected to this switch, i see traffic for entire VLAN on wireshark. I

am not suppose to see traffic between other hosts/server since this server port is not

configured as a span port. It seems like 6500 switch is acting as a hub . There is no span

session configured on this switch. Our system admins have been complaining about

performance issue on their servers. I used wireshark on different servers on that switch

and i see same result. All the servers connected to this switch is seeing traffic for

entire vlan regardless of ports. I even connected my laptop to this switch and i can see

traffic for entire vlan.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
xcz504d1114 Wed, 01/07/2009 - 14:10
User Badges:
  • Bronze, 100 points or more

Switches will flood packets in which they don't know the destination to, out every port except their originating port, so you will always see some unicast traffic. Every 5 minutes by default the mac-address entries will age out, if it is excessive and you are sure you are actually seeing what you're claiming, check the mac-address tables to make sure there are entries being populated (show mac-address-table) and you can even adjust the default mac-address aging timers to a higher value.


Is there just 1 6509? Is there any HSRP? How many packets per second are you seeing on the port with wireshark? With wireshark, view the IO graph for a 10 minute capture to see if you are having huge packet bursts. Are the servers using dual NIC's? Do they have their NIC teaming setup properly (I see it all the time in my datacenter, inproper NIC teaming can cause mac-address issues, check your logs to see if you have mac flapping, that is a dead give away)?


Is there any multicast / broadcast traffic or is it all unicast?


Craig

altaf007 Wed, 01/07/2009 - 14:19
User Badges:

Hi Craig,

I see ip traffic, broadcast, eigrp update, etc between different hosts/server. I see their web, e-mail, http,ssh, telnet, etc.


Is there just 1 6509? We have two 6509 connected with layer two etherchannel.They both have same VLAN/subnet for servers


Is there any HSRP? yes,I have different vlan for HSRP. This vlan has different subnet.


How many packets per second are you seeing on the port with wireshark? I am seeing 20 to 50 mbps packets


Are the servers using dual NIC's? No dual nic. just single 1 gig NIC


Do they have their NIC teaming setup properly (I see it all the time in my datacenter, inproper NIC teaming can cause mac-address issues, check your logs to see if you have mac flapping, that is a dead give away)? I already checked that and i did not see anything in logs.

I

xcz504d1114 Wed, 01/07/2009 - 14:31
User Badges:
  • Bronze, 100 points or more

Kwu gave you a link to exactly what I was thinking about with the HSRP configurations (Asymmetric routing). One of the characteristics is large packet bursts at random intervals. There are 2 ways to over come this, one as recommended in documentation is to adjust the mac-address agining timer from 5 minutes (300 seconds) to 4 hours (14400 seconds) to match the ARP aging. Another method is to not manually load balance between your HSRP gateways, have one router be the default gateway for everything with the second router being the standby for everything.


Every port you will see broadcasts, EIGRP, and occaisionally you will see the other stuff. How often you see the other stuff is the concern.


To clarify, you are seeing 20 to 50 mbps? Is that Megabits per second? Million packets per second? I'm looking more for a count of how many packets per second you are seeing, not necessarily the bandwidth usage. But even 50 Mbps (Megabits per second) on a 1 Gigabit connection is insignificant. Now some NIC's handle "junk" traffic, I consider "junk" anything that is not relevant to that device that is recieving it, that is why I am trying to get an idea of how many packets per second you are seeing.


Craig

Yudong Wu Wed, 01/07/2009 - 14:15
User Badges:
  • Gold, 750 points or more

Not sure if the issue is related to unicast flooding.

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801d0808.shtml


Also please verify with Server guys about:

1. how many actived NIC card on each server and how they are connected to switch.

2. If server has multiple NIC card, does it use one to send/receive traffic. You might aslo sniffer the incoming traffic on the port which the server is connected to to find out this.


altaf007 Wed, 01/07/2009 - 14:50
User Badges:

There is only one NIC 1gig on each server. I tried to configure 6500 switch with following command but it does not work

unicast flood protection. I am gonna open case with cisco TAC about this.

Giuseppe Larosa Wed, 01/07/2009 - 15:14
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Altaf,

it looks like the CAM table is full.


check with


sh mac-address-table count


the max is in the order of 65,000 MAC addresses


Last summer we had a strange problem caused by CSM that was creating random MAC addresses when the traffic volume was high (more then 2 Gbps) and high number of connections.

We solved this with a CSM firmware upgrade.


Are you using CSM and firmware is 2.3.x or similar ?


We now have 4.2(9) on them.

We opened a service request and they found the problem hitting a known bug very quickly.



Hope to help

Giuseppe


Actions

This Discussion