SG300 dropping packets

Brian Bergin · ‎04-11-2012

We have a customer who we sent to Cisco to replace some aging Dell switches. They purchased 5 SG300-52’s for 2 different networks. Their production LAN has 2 “live” switches and 1 spare. The 2^nd, a development LAN has 1 switch and 1 spare. Their primary production SG300-52 has GE1-8 VLAN’d off as VLAN2 for public IPs. The untrusted (WAN) interfaces of 2 x ASA-5510’s, 1 x ASA-5505, and 2 x RV082 v2’s are connected to GE2-6. GE1 is the uplink to the co-location center’s Cisco switches. GE7 & 8 are spare ports. Each SG and device port is hard coded for 100/Full.

One of the ASA-5510’s and the ASA-5505 maintain a site-to-site VPN (the development LAN used to be in a different facility hence the VPN). Recently the developers have stated the performance is horrible. I noticed ping traffic loss from PCs on the dev side to servers on the production side in the order of 20-30%. I assumed it was a VPN issue so I opened a ticket with Enterprise TAC (all the ASAs and the SGs have either SmartNet or extended support contracts). TAC determined the problem happened even if you ping from inside the ASA to the untrusted side of the other ASA thus eliminating the VPN as the culprit.

The 2^nd ASA-5510 has the AIP module and was not even live until this weekend. Turning it up and giving it a basic config returned the same results. #ping x.x.x.x repeat 100 will drop 20-40 packets. I have no security enabled on the SGs and even tried using the spare SG300-52 this weekend in place of their primary with the same result. I’m to the point of returning one of the Dell switches to production, but this cannot be a good sign. I’m also a bit frustrated that I’ve yet to figure out how to get Cisco Enterprise to speak with Cisco Small Business on this. The customer has over $10k invested in Cisco equipment and Cisco isn’t jumping in to figure this out.

The latest rep wants a packet capture from the SG300’s VLAN2 but there are no PCs there to do this with and the manual doesn’t even talk about doing this. Can anyone suggest how we can do this as well as get the 2 divisions working together to fix this? BTW, the RV082’s exhibit the SAME exact problem. I can ping from ANY device on VLAN2 to any other device and drop packets. Copying a simple 1MB file over the VPN can take minutes where it should take 1 second. I can reproduce this for anyone 24/7.

Thoughts?

rocater · ‎04-12-2012

Hi Brian,

The issue you describe really would be best served with a packet capture so we could see what is going on. Before proceeding though, please make sure the switch has been updated to the latest firmware available. It would also help to enable logging on the switch as it may give an idea as to the problem. The SG300 does offer 'port' mirroring. In this case the 'port' you could mirror is all of vlan2 and do it from any other port on the switch. For a packet capture go to Administration > Diagnostics > Port and VLAN Mirroring.

With regard to involving both TAC and SBSC, it is often easiest to get a TAC engineer online and then call the SBSC. We here in the SBSC are more on-demand and can be reached anytime.

If you are able to get a packet capture, we can review it to see what is happening.

Brian Bergin · ‎04-12-2012

Robert,

Are you saying that from a machine, say 10.0.0.180 connected to VLAN1 I can install Wireshark and set the SG300 to mirror the entire VLAN2 to the port that 10.0.0.0.180 connects to? These are all Dell PoweEdge servers with Dell Remote Access Cards so even if the NIC on 10.0.0.180 drops normal traffic I can still get to the console remotely (these boxes are all in an unmanaged data center).

Randy,

Yes, I can get packet captures from any of the ASAs as well if that's better than the above solution. In fact, it's be easier to get those captures as none of the servers are connected to the SG300-52 that the ASAs are so I'd have to have someone go there and move a cable (not a problem, just a bit more coordination I need to do).

Thanks guys...

Brian

rocater · ‎04-13-2012

That is correct. The switch will let you mirror all traffic from a vlan to any other port you select. If possible all you would have to do is bring in a laptop and attach to one of the port and setup the mirror.

As Randy said though, a capture from the ASAs may help as well. It should show traffic recieved from both sides. But before doing all this, just double check the firmware. The latest version at this time is 1.1.2.0.

rmanthey · ‎04-12-2012

Brian,

The switch it sounds like it is just doing layer 2 no, layer 3 routes correct?

What firmware are you on? There is a known issue in one of the older versions where the MAC table has problems.

Have you tried to set the auto negotiation to auto and does that make any difference?

We might need to look deeper into the issue with a wirecapture. You stated you don't have a pc at that location to capture correct? Is there any way to do a capture on one of the ASA's?

Cisco Small Business Support Center

Randy Manthey

CCNA, CCNA - Security

Brian Bergin · ‎04-13-2012

Yes, was on 1.1.2.0 five minutes after opening the boxes. I'll work on getting captures and get back to you.

Brian Bergin · ‎04-15-2012

I pulled about a 400MB VLAN2 -> Wireshark capture by connecting a 2008 R2 server to the SG300-52 on port GE42 and mirroring VLAN2 (contains the untrusted interfaces of the ASAs and RV082s as well as the uplink to the co-location center's Cisco switches). 7Zip got it down to 165MB. What's the next step with the capture?

Thanks...

rocater · ‎04-16-2012

Hi Brian,

In the packet capture you would want to look for traffic you know should be there but is not. An example would be a constant ping from one side to the other that is not over the tunnel. If possible, I would suggest calling the support center. We would want to look over the switch settings and the packet capture.

Brian Bergin · ‎07-12-2012

Just to close this out. The issue was NOT the SG300's but in fact the hosting company's routing tables. Since the LANs are on different subnets but have a shared port in an upstream Cisco switch data would leave one LAN, make a u-turn at the Cisco switch, and return to the other LAN to traverse the VPN. This had worked for many years and only started having problems a few months after swapping out the Dell switches, but in the end we finally proved it was the hosting company's problem when e-mail would not deliver to another IP owned by the same hosting company so routing was not just a problem for the 2 LANs but for many. After a few days of banter back and forth they got with Cisco, got their ducks in a row, and solved it!

During this time, we solved this with a better internal solution. We added an SG300-10 in Layer 3 mode and are now doing full gigabit speeds between the 2 LANs with the VPN out of the equation.