Packet loss in VLAN setup

Gerald Vogt · ‎09-07-2008

I have a Cisco 1812 router which currently provides internet for an office LAN. FE0 connects to the internet. FE1 (the second WAN port of the router) connects to a LAN switch to which all LAN devices are connected to. This works fine so far.

I now want to transition the LAN connection from FE1 to one of the LAN ports on the Cisco in order to utilize VLANs and other functions.

The configuration of FE1 before the transition is:

interface FastEthernet1

ip address 10.0.0.254 255.255.255.0

ip broadcast-address 10.0.0.255

ip nat inside

ip virtual-reassembly

duplex auto

speed auto

no cdp enable

end

Now I thought I simply setup VLAN 10 the same way, take FE1 down, and add FE9 in access mode to VLAN 10, unplugging FE1 and plugging the same cable into FE9.

Configuration changes:

!

interface FastEthernet1

no ip address

no ip broadcast-address

shutdown

exit

interface vlan 10

ip address 10.0.0.254 255.255.255.0

ip broadcast-address 10.0.0.255

ip nat inside

ip virtual-assembly

exit

interface FastEthernet 2

switchport access vlan 10

exit

interface FastEthernet 9

switchport access vlan 10

end

Then I plug the cord from FE1 to FE9. If I connect a computer to FE2 I don't have issues. Everything works fine.

However, all devices connected the LAN switch have problems. If I run a "ping 10.0.0.254" on any of those computers I see a packet loss of about 50-60%. I tried different cables and different ports. It does not seem to be a simple hardware issue. When I enable "debug ip icmp" I can see that the router only infrequently sends the replies. It seems as either the incoming ICMP packet on the router gets lost or not processed correctly. pings from the router to devices in the LAN, e.g. "ping 10.0.0.35" seem to work fine. No loss reported.

What could possibly cause this packet loss?

Calin C. · ‎09-08-2008

Hello!

I don't understand where is the ports FE2 and FE9 (a switch connected to FE1 on the router maybe) ?

Please clarify!

Thanks!

Cheers,

Calin

Gerald Vogt · ‎09-08-2008

The 1812 has two WAN ports FE0 and FE1. It has 8 LAN ports FE2 to FE9.

Currently, FE1 connects to the main LAN switch (a simple managed Planex switch). In the future I want to run the Planex into the FE9 port of the 1812 so that I am able to start using the capabilities of the LAN switch in the 1812.

FE2 is just another LAN port which I have assigned to the same VLAN on the 1812 for testing.

Calin C. · ‎09-08-2008

"Then I plug the cord from FE1 to FE9"

You did a loop on the router. Why?

The idea, as I see it is like this. You have 2 WAN interfaces, one primary, one backup (or load sharing or whatever you want) but this remains your WAN connections.

You configure a "interface vlan 10" (it's OK how you did it) and then FE9 configure as a trunk that allow carrying VLAN 10. You connect FE9 to the switch (Planex) and in this you connect your clients. Another approach would be to configure the LAN ports on FE1 as access ports and connect there directly clients. Or you can mixt the 2 methods above, but do not connect a switch into a access port.

Tell me if it is working like this!

Good luck!

Cheers,

Calin

Gerald Vogt · ‎09-08-2008

I unplug the cable from FE1 and plug that same cable into FE9.

The person who set this router up used the WAN port FE1 for the LAN connection. I want to change that.

The Planex switch is managed, but currently only port-pased. The LAN only consists of a single VLAN at the moment.

FE1 is a WAN port. It cannot be configured for VLANs. The LAN switch in the 1812 consists only of ports FE2-FE9.

Giuseppe Larosa · ‎09-08-2008

Hello Gerald,

FE1 is a router port and can be connected to a switch with a straigh cable.

Later, you plug the same cable into FE9 that is part of an etherswitch module: in old times connecting two switch ports required a cross-over cable.

I would try to use a cross-over cable.

Hope to help

Giuseppe

Gerald Vogt · ‎09-08-2008

Hi Guiseppe,

both router and Planex switch are not really from the old times. Both are auto-sensing cross-cables. I thought if the auto-sensing does not work there would be no connection at all and not a 75% packet loss in one direction only. I'll see if I find a cross-cable anyway. Haven't used those for a long time. I'll also try to put a different unmanaged switch into the connection to test whether this helps.

Thanks, Gerald

Gerald Vogt · ‎09-08-2008

Some additional information, making the whole thing even more confusing to me:

When the Planex is connected to the FE9 port as I want it to be.

* pings from a computer 10.0.0.35 to the cisco 10.0.0.254 have 75% packet loss.

* each packet which goes through shows up in "debug ip icmp" on the cisco.

* if I ping the computer on the cisco (ping 10.0.0.35 repeat 1000) I have no packet loss at all.

* I currently connect through the internet and ssh to the cisco (FE0/Dialer0). Another interesting observation is whenever the Cisco gets some of the ICMP echo requests from the computer the SSH session will hang temporarily. If the computer happens to successfully send 5 echo requests in a row to the cisco the SSH session basically hangs for 5 seconds. As long as the pings fail the SSH session is fine. So it seems as if those pings make it to the router but even though they don't appear in the 'debug ip icmp' output.

Calin C. · ‎09-08-2008

Hi Gerald!

Do this simple test. Connect one PC to a port (FE8 for example) and ping the Cisco router 10.0.0.254. Do you have any packet loss?

Configure FE8 as trunk that is permitting VLAN 10, connect the Planex to it, connect the PC to Planex and ping the 10.0.0.254. Do you have packet loss?

If yes, then is something weird with Planex switch. Try to switch it with another sw, different brand, if it is possible.If not, then your FE9 on the Cisco has a problem.

Good luck!

Gerald Vogt · ‎09-08-2008

Thanks calin.

If I connect a PC to FE2 I can ping the router without problems. Connecting the Planex to any LAN port FE2-FE9 on the Cisco always causes problems.

I cannot configure a port in trunk mode as the Planex is not configured for 802.1q. It is configured for port-based VLANs only. I simply connect an access mode port of the Cisco to a port which belongs to the LAN VLAN on the Planex.

Tomorrow I'll try to put a different switch into the conection to see if that changes anything. However, I am not fully convinced that it is the Planex causing those issues. Traffic seems to pass fine if it is initiated on the Cisco (the ping from the Cisco router to a computer connected on the Planex has not packet loss).

Moreover, if the pings are initiated from the computer on the Planex they seem to arrive on the Cisco (the SSH session from the internet hangs, although it is a completely different interface). How would an ethernet frame arriving on the FE9/VLAN interface of the Cisco cause a hanging SSH/TCP session? Doesn't this mean that the frames arrive on the Cisco and are processed somewhere but then cause some issue??

I don't see why a ICMP echo reply goes through fine but a ICMP echo request not...

Gerald Vogt · ‎09-09-2008

I tested a few things today.

* The cable does not matter. Cross or straight. Same results.

* Changing ports (FE2-FE9 or any Planex port) does not make a difference.

* I've also put in a different brand unmanaged switch between the FE9 port and the Planex. No difference. Pings through the Planex, unmanaged switch to the Cisco have large packet loss. Pings through the unmanaged switch only have no packet loss just like a computer directly connected to the Cisco.

* pinging the Cisco initiated on a computer connected to the Planex will show packet loss of 60% or worse.

* pinging a computer connected to the Planex from the Cisco results on no packet loss.

Now I am fully stumped. The Planex works fine while connected to the FE1 WAN port. It does not work fine while connected to the FE9 LAN port. I does not seem to be an electric problem as the same appears if an unmanaged switch is put in between. So I think it is safe to rule out a basic physical layer issue.

It is bizarre. Frames from the Planex to the Cisco go through just fine as long as it is a reply. Requests from the Planex to the Cisco have packet loss. The frames still seem to arrive on the Cisco as they affect other connections, e.g. make the SSH connection from the internet to the Cisco router hang for a few moments. For whatever reasons, it causes problems when it arrives on a LAN port while it is fine when it arrives on the WAN port.

Gerald Vogt · ‎09-10-2008

I think I have found the reason: The Cisco uses the MAC address of FE0 for the VLAN SVIs. In other words the vlan interface has the same hardware address as the FE0 WAN interface.

Now, the setup in this company uses the Planex managed switch with port-based vlans for WAN and LAN connections. One vlan is used for all the LAN devices. The other vlan is used to connect multiple internet routers (the Cisco is only one of them) with the ONU.

Thus the situation is that the Planex will find the identical hardware address on one port in the WAN VLAN and on one port on the LAN VLAN. This seems to confuse it.

I think the observed issues correlate with this: pings from the Cisco into the LAN work because the switch immediately learns the hardware address on the LAN port and accepts the reply which arrives quickly afterwards.

The opposite direction does not work most of the time: as the cisco is hardly sending anything into the LAN (at least during these tests) the switch assigns the hardware address to the WAN port. As it is a different VLAN frames for the cisco are not accepted through LAN except some random moments when the cisco did actually send something into the LAN before.

This would also explain why the few successful pings in the LAN to the vlan interface correlate with a hanging SSH session through the internet port: during that time the hardware address is used on the LAN side and not on the WAN side thus the internet connection is not available for a moment.

Well, took me some time going through some larger packet dumps until I have noticed this oddity in the Cisco. I have tried to change the hardware address of the vlan interface but that's not possible. Seems as if I have to change the FE0 hardware address. But I will test this later when people don't need the internet connection. Tomorrow I'll now.

Does anyone have the Cisco uses the FE0 hardware address for the vlan interfaces? I thought it would be better if they have used an unused hardware address for the vlans. Is there a way to see all the used hardware addresses in an IOS router in a single list?

It seems to me as if the best solution would be to take the WAN ports from the Planex and use a completely different unmanaged switch instead. Same result but no confusion anymore.

Thanks for all the people you posted and tried to help!!

Giuseppe Larosa · ‎09-10-2008

Hello Gerald,

very interesting information.

You could move the WAN interface to FE1 to see if this solves this issue: if MAC address for SVI VLans is still taken from FE0 even if now unused, this should solve the issue.

Probably the router takes the first FE to give a MAC to the SVI interfaces.

to see MAC addresses in use you can do sh ip arp and look for entries that have a - sign in the expiration time this says it is a router's ip address and MAC pair.

Hope to help

Giuseppe