Switching Issue

[thiyagarajankalaiselvan]

Dear Experts,

I have been experiancing a critical issue and I am in need of your support to find out the cause.

The issue is, My Backup servers and few other production server are in the same VLAN.

Whenever Backup jobs are running I could see lot of packets drops on other production servers which are in the same VLAN.

When backup jobs are stoped, there was no issue. During the issue when I ping the default gateway of the VLAN there was no issue.

Can anybody help me to address this issue?

Is there any chance to identify the utilization of VLAN?

I am wondering how Backup server jobs cause packet drop of other servers in the same VLAN.?

is there any command to isolate and fix this challenging issue?

If anybody needs any command output. I will share it.

Any help would be appreciable.

Regards,

Thiyagarajan

[danrya]

Can you please supply some  additional information so we can help?

What model switch and what version of IOS?  These can be found in the "show version".  Can you do a "show interface" on the backup server and one of the servers that's dropping packets, as well as on the VLAN interface?  What is being backed up by the backup server when you see the packet drops?

Dan

[thiyagarajankalaiselvan]

Please find the details of the Switch and one of the production server connecting Interface on which we are experiancing packet drop issue.

cisco WS-C6509-E (R7000) processor (revision 1.2)

GigabitEthernet0/11 is up, line protocol is up (connected)
   Description: ***Production server where we are having packet drop issue***
   5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 45029000 bits/sec, 3905 packets/sec

The only thing I observe is that during the time of issue lot of traffic is flowing toward the server.

Any hints to fix the issue would be appreciable.

Regards,

T.K

[Jon Marshall]

thiyagarajankalaiselvan wrote:
Please find the details of the Switch and one of the production server connecting Interface on which we are experiancing packet drop issue.
cisco WS-C6509-E (R7000) processor (revision 1.2)
GigabitEthernet0/11 is up, line protocol is up (connected)
   Description: ***Production server where we are having packet drop issue***
   5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 45029000 bits/sec, 3905 packets/sec
The only thing I observe is that during the time of issue lot of traffic is flowing toward the server.
Any hints to fix the issue would be appreciable.
Regards,
T.K

T.K

It could be a number of things.  2 things spring to mind -

1) what modules are the servers connected into ie. are they connected to the same linecard or are they spread across linecards ?  What supervisor are you running ? note if it is a supervisor 2 are you running an SFM (Switch Fabric Module) in slot 5.

I ask because it may be that the modules the servers are connected into could only be classic linecards or fabric enabled but only using the shared bus. This would mean with a lot of backup traffic the shared bus could be getting overloaded and hence you are seeing packet drops.

2) If you are running fabric enabled linecards such as the WS-X6548-GE-TX with a sup720 or a sup2/SFM the linecards would have access to the switch fabric but the above linecard for example can only support 1Gbps per 8 ports so again because of the amount of backup traffic you could be dropping packets to some servers. It all depends on how the servers are spread across the linecard.

If you are running fabric only linecards such as the WS-X6748-GE-TX with a sup720 then it is unlikely you will be getting the same issues as above as these modules have a much greater connection to the switch fabric.

So basically need to know the sup/modules and spread of servers, both backup and non-backup across those modules.

Jon

[vdadlaney]

Hi,

Can you provide all the details on your switch with regards to what kind of line cards are installed and what slots as well as what kind of sup's are you using and the IOS version.

The output that you provided for that server unless I missed something does not indicate anything unusual since its only 45Mb output on a Gig port and that too on a 5 minute interval. You might want to change the load interval under the interface to 30 seconds to get a better idea.

In addtion you do mention that the Production servers and the Backup servers are on the same VLAN however could you double check to see if any servers that are not on the same vlan are backing up to your backup server and if so you might want to check for unicast flooding. We had a similar issue which turned out to be unicast flooding as what was happening was during the interval when the CAM entry had aged out the traffic though destined for a backup server was getting flooded to all hosts in that VLAN. You could put a sniffer (or just run wireshar if you are allowed to on any production server during the high utilization period) and check to see if you are seeing any traffic between other hosts on this server which would indicate unicast flooding.

You might also want to confirm if there are any monitor sessions (Span) configured and if so remove them to see if they fix the issue. Depending on the code I believe that TX span session might result in flooding. Thx