ATM xconnects packet loss across EoMPLS 200meg connection

[David Williams]

I have a strange situation I'd like to run by the community.  We have recently acquired a 200 meg ethernet connection to replace our OC3 between two of our major markets.  Both connections are running mpls.  The OC3 has an MTU of 4470 of course and the Ethernet is set to 9000.  We tested the Ethernet circuit last week and everything looked good.  There are no errors on either end of the connection.  No packet loss or latency.  Both MPLS VPN and MPLS traffic function great.

The problem has to do with a handful of ATM xconnects that we have built between two PE routers that run across that intermarket link.  When we roll traffic to the Ethernet connection we take 50% packet loss.  When we roll back to the OC3 the packet loss clears up.  The only traffic we know of that is affected is the ATM xconnects.  The one thing we found out that we didn't know was that the ethernet connection is actually an EoMPLS connection through our carriers network.  I'm not sure how that would impact things.

One end of the ethernet connection is plugged into a Gig switchport on a switching blade with IP term on a vlan SVI in a 7606 Sup 720 3BXL.  The other end is on a SUP720 3BXL gig subint on a 7603.  Another interesting point.  The loss only seems to occur on traffic from the 7603 to the 7606.  Traffic from the 7606 to the 7603 seems to work fine.

Any ideas?

[Edison Ortiz]

The one thing we found out that we didn't know was that the ethernet
connection is actually an EoMPLS connection through our carriers
network.  I'm not sure how that would impact things.

It will definitely impact you.

If their MTU isn't large enough, it does not matter how large you set your MTU, they will drop the packet as EoMPLS do not support fragmentation and reassembly.

[David Williams]

Their MTU is maxed out at 9216.  We can pass 9000 byte ping packets with the df-bit set.  Aside from that, we are not doing any cell packing so the ATM ATOM labled cells would be quite small, and from what I understand, would never approach even a low MTU.

ROUTER#ping X.X.X.X size 9000 df-bit

Type escape sequence to abort.
Sending 5, 9000-byte ICMP Echos to X.X.X.X, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/12/16 ms

[Edison Ortiz]

Cool

Since they are giving you a 200MG on a 1Gb handoff, you must create a shaper to instruct your flows, the pipe is not 1Gbps, it is a 200Mbps.

You mentioned having a 7600 router, so you need a SIP/SPA module.

[David Williams]

Well, we are not even close to peaking to that 200 meg.  In fact we are less than 50% under normal load right now.  Why would we need an Ethernet SIP and SPA to shape?  Is shaping not supported on the WS-X6724-SFP? or on the Gig port on the SUP720?

Why would traffic pass fine in one direction and not the other?  Keep in mind.  The ONLY affected traffic is ATM xconnect traffic which is why we didn't catch it in testing.  The xconnects are built on the two PE routers so all the P routers should be doing is passing it on.

PE---->P----------(carrier PE)---New Ethernet----(carrier PE)---->P---------->PE

[Edison Ortiz · Accepted solution]

Is shaping not supported on the WS-X6724-SFP? or on the Gig port on the SUP720?

No and No. You need a WAN module for shaping..

Why would traffic pass fine in one direction and not the other?

Very hard to troubleshoot on an internet forum. I recommend opening a ticket with TAC and the provider to find the real issue..

[David Williams]

Yeah we've already opened a TAC case.  We'll see where that goes.  Thanks