Very weird packet loss / lag issue on vlan

Unanswered Question
Apr 30th, 2008
User Badges:

Hello,


We are experiencing a very bizare issue. We have a vlan 169, it has for config


ip address XXX.XXX.XXX.XXX 255.255.255.224


Everything works very well and as soon as we add another IP as a secondary ip for that vlan interface, it starts to time out on pings and the MS are higher. Usually pings at 10 MS, then starts going up to 50 even 100 MS and times out several times.


We were running on many secondary ip's on that vlan for a while and just now it started acting up.


We are not sure if there is some kind of loop or what can be causing this issue..


Can you please help out with this?


Thank You

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
interwebmedia Wed, 04/30/2008 - 16:25
User Badges:

We just conducted a bit more tests and we realized the following...


We have over 100 active vlans and configured over 400 (in waiting to install more servers). MANY of those vlans have secondary ip addresses as that is how we assign them when customers order more IP's.


What we are realizing is that, many if not all of the customers on the vlans which have secondary IP addresses are experience packet loss and lags...


As a test on one vlan, we removed ALL the secondary ip addresses leaving only the main one, and everything works beautifully...


Can someone please guide us into what is probably happening?


Thanks

mlitka Fri, 05/02/2008 - 05:41
User Badges:

What is your switch model and IOS version? Have you checked the Bug Toolkit? How does the interface utilization look? It maybe helpful to SPAN one of the those ports and get a packet capture with Wireshark. Perhaps there is a broadcast storm or some type of routing or switching issue with the secondary addresses that is black holing traffic.


Just a few ideas. Let us know how it goes.

interwebmedia Fri, 05/02/2008 - 10:18
User Badges:

Switch model is Cisco 3550 48 Port - EMI


IOS 12.2


What do you mean by BUG TOOLKIT?? I am not sure what you are referring to.


I did a span of all the ports and checked and there does not seem to be anything wrong unless the Wireshark is dropping packets in order to process everything it is seeing but I am not sure.


As soon as I remove all the secondaries from a VLAN and route them using IP ROUTE there is no longer packet loss or lags on that specific VLAN. Very weird issue.


Thank you for your help, please let us know


Very appreciated

interwebmedia Fri, 05/02/2008 - 11:11
User Badges:

Thanks. But is it known that having too many secondary addresses can cause lags and packet loss??


We also noticed that for the vlans in the range of 1-50 there does not seem to be a problem, as if the first vlans it's fine, but the customers on vlans 150 and + and have secondary addresses configured seems to be the ones complaining..


Does it make any sense?


Thanks

mlitka Fri, 05/02/2008 - 11:22
User Badges:

It makes sense. I am not sure if that is a known bug or not, hence why I would suggest looking at the bug tool kit. Are your interfaces getting slammed? What is the reason for all these secondary addresses? You may be having issues with too large a broadcast domain.

interwebmedia Fri, 05/02/2008 - 11:54
User Badges:

Hello,


ok I checked the bug toolkit, there was one of the bug that resembled so in any case, we just did the software upgrade but error is still here.


The reason for secondary IP addresses is because we assign ONE vlan per dedicated server and we assign additional IP's to that customer using a secondary IP address on his Vlan.


Yesterday we did a quick test by removing ALL the secondary addresses and then the switch REALLY lagged, even telnet was slow. Not sure if it is because people were actively using those IP's and now they did not exist anymore or something like that but it was even worse.


Can you suggest anything else for me to look at or something?


Thanks

mlitka Fri, 05/02/2008 - 11:58
User Badges:

Can you diagram this? It may make more sense seeing it visually. You may want to call TAC if you can on this one though.

interwebmedia Mon, 05/05/2008 - 09:57
User Badges:

Hello,


What do you mean by diagram this exactly?


Diagram of what?


On another part, have you ever seen a problem with too many secondary addresses on different interfaces etc... ?


Thanks

mlitka Mon, 05/05/2008 - 10:00
User Badges:

It would be helpful to see a network diagram of your environment so we can see how this is setup.


I have not seen or heard of this issue before.

interwebmedia Mon, 05/05/2008 - 11:17
User Badges:

Well here is an explanation of it:


--- CISCO 3550 48 PORT EMI ---- as our core.


Each fastethernet port is connect to an uplink to cisco 2950 switches, then the ports on those switches are going to the servers.


We operate on ONE vlan per customer, therefore each dedicated customer has its own vlan.


When they want additional IP's, we do a secondary ip address on their vlan and assign it to them like that.


Some vlans are used for like shared hosting, so multiple ports on the 3550 and the 2950's sometime use the same vlan as they serve the same range of IP's (since it is different servers but all on for example the same Class C).


Hope this gives you a better idea.


Thanks

Actions

This Discussion