[SOLVED] 7k to 7k packet-loss on peer-keepalive

andreas.wassatsch · ‎07-14-2015

Hello, I have two N7706 with 48-port modules in slots 2 + 5, and have build a L3 port-channel using ports e2/1 and e5/1 each. Ports are connected with new SFP-H10GB-CU3M cables.

ping'ing across this port-channel shows 0.3 - 50% packet loss (depending on packet size). Any obvious errors in my config, or any explanation for this packet loss ? (loss pattern is same if I shut either e2/1 or e5/1 on a side)

Best Rgds,

Andreas

PS: Systems just arrived, have no smartnet ID yet - and our distributor hasn't managed to get TAC involved yet..

N7706-A

===================

feature lacp

!

interface port-channel11

speed 10000

ip address 10.64.255.17/30

!

interface Ethernet2/1

speed 10000

channel-group 11

no shutdown

!

interface Ethernet5/1

speed 10000

channel-group 11

no shutdown

N7706-B

===================

feature lacp

!

interface port-channel11

speed 10000

ip address 10.64.255.18/30

!

interface Ethernet2/1

speed 10000

channel-group 11

no shutdown

!

interface Ethernet5/1

speed 10000

channel-group 11

no shutdown

N7706-A# ping 10.64.255.18 vrf default packet-size 1500 count 50
PING 10.64.255.18 (10.64.255.18): 1500 data bytes
1508 bytes from 10.64.255.18: icmp_seq=0 ttl=254 time=2.043 ms
1508 bytes from 10.64.255.18: icmp_seq=1 ttl=254 time=2.057 ms
1508 bytes from 10.64.255.18: icmp_seq=2 ttl=254 time=1.636 ms
1508 bytes from 10.64.255.18: icmp_seq=3 ttl=254 time=1.527 ms
1508 bytes from 10.64.255.18: icmp_seq=4 ttl=254 time=1.595 ms
1508 bytes from 10.64.255.18: icmp_seq=5 ttl=254 time=1.561 ms
1508 bytes from 10.64.255.18: icmp_seq=6 ttl=254 time=1.675 ms
1508 bytes from 10.64.255.18: icmp_seq=7 ttl=254 time=1.617 ms
1508 bytes from 10.64.255.18: icmp_seq=8 ttl=254 time=1.67 ms
1508 bytes from 10.64.255.18: icmp_seq=9 ttl=254 time=1.524 ms
Request 10 timed out
1508 bytes from 10.64.255.18: icmp_seq=11 ttl=254 time=1.729 ms
1508 bytes from 10.64.255.18: icmp_seq=12 ttl=254 time=1.581 ms
1508 bytes from 10.64.255.18: icmp_seq=13 ttl=254 time=1.604 ms
1508 bytes from 10.64.255.18: icmp_seq=14 ttl=254 time=1.567 ms
1508 bytes from 10.64.255.18: icmp_seq=15 ttl=254 time=1.575 ms
1508 bytes from 10.64.255.18: icmp_seq=16 ttl=254 time=2.126 ms
1508 bytes from 10.64.255.18: icmp_seq=17 ttl=254 time=1.679 ms
1508 bytes from 10.64.255.18: icmp_seq=18 ttl=254 time=1.58 ms
1508 bytes from 10.64.255.18: icmp_seq=19 ttl=254 time=1.527 ms
1508 bytes from 10.64.255.18: icmp_seq=20 ttl=254 time=1.523 ms
Request 21 timed out
1508 bytes from 10.64.255.18: icmp_seq=22 ttl=254 time=1.57 ms
1508 bytes from 10.64.255.18: icmp_seq=23 ttl=254 time=1.551 ms
1508 bytes from 10.64.255.18: icmp_seq=24 ttl=254 time=1.579 ms
1508 bytes from 10.64.255.18: icmp_seq=25 ttl=254 time=1.548 ms
1508 bytes from 10.64.255.18: icmp_seq=26 ttl=254 time=1.548 ms
1508 bytes from 10.64.255.18: icmp_seq=27 ttl=254 time=1.553 ms
1508 bytes from 10.64.255.18: icmp_seq=28 ttl=254 time=1.535 ms
1508 bytes from 10.64.255.18: icmp_seq=29 ttl=254 time=1.599 ms
1508 bytes from 10.64.255.18: icmp_seq=30 ttl=254 time=1.552 ms
1508 bytes from 10.64.255.18: icmp_seq=31 ttl=254 time=1.572 ms
Request 32 timed out
1508 bytes from 10.64.255.18: icmp_seq=33 ttl=254 time=1.588 ms
1508 bytes from 10.64.255.18: icmp_seq=34 ttl=254 time=1.701 ms
1508 bytes from 10.64.255.18: icmp_seq=35 ttl=254 time=1.681 ms
1508 bytes from 10.64.255.18: icmp_seq=36 ttl=254 time=1.624 ms
1508 bytes from 10.64.255.18: icmp_seq=37 ttl=254 time=1.623 ms
1508 bytes from 10.64.255.18: icmp_seq=38 ttl=254 time=1.667 ms
1508 bytes from 10.64.255.18: icmp_seq=39 ttl=254 time=1.691 ms
1508 bytes from 10.64.255.18: icmp_seq=40 ttl=254 time=1.618 ms
1508 bytes from 10.64.255.18: icmp_seq=41 ttl=254 time=1.616 ms
1508 bytes from 10.64.255.18: icmp_seq=42 ttl=254 time=1.629 ms
Request 43 timed out
1508 bytes from 10.64.255.18: icmp_seq=44 ttl=254 time=1.723 ms
1508 bytes from 10.64.255.18: icmp_seq=45 ttl=254 time=1.602 ms
1508 bytes from 10.64.255.18: icmp_seq=46 ttl=254 time=1.654 ms
1508 bytes from 10.64.255.18: icmp_seq=47 ttl=254 time=1.63 ms
1508 bytes from 10.64.255.18: icmp_seq=48 ttl=254 time=1.612 ms
1508 bytes from 10.64.255.18: icmp_seq=49 ttl=254 time=1.638 ms

--- 10.64.255.18 ping statistics ---
50 packets transmitted, 46 packets received, 8.00% packet loss
round-trip min/avg/max = 1.523/1.636/2.126 ms

Reza Sharifi · ‎07-14-2015

Hi,

Do you have the same issue if you don't use Portchannel at all? Just use one 10gig layer-3 connection and test again.

HTH

andreas.wassatsch · ‎07-14-2015

Hello,

thanks for your reply - yes, this happens even with only one int in the portchannel, or with direct links.

I've just found the cause, we had "copp profile strict" in place which polices icmp down to a very low rate (130 kbps), much too less for the ping tests..

#show policy-map interface control-plane

    class-map copp-system-p-class-monitoring (match-any)
      match access-group name copp-system-p-acl-icmp
      match access-group name copp-system-p-acl-icmp6
      match access-group name copp-system-p-acl-mpls-oam
      match access-group name copp-system-p-acl-traceroute
      match access-group name copp-system-p-acl-http-response
      match access-group name copp-system-p-acl-smtp-response
      match access-group name copp-system-p-acl-http6-response
      match access-group name copp-system-p-acl-smtp6-response
      set cos 1
      police cir 130 kbps bc 1500 ms
        conform action: transmit
        violate action: drop
      module 1:
        conformed 8293884 bytes,
          5-min offered rate 12928 bytes/sec
          peak rate 14206 bytes/sec at Tue Jul 14 03:13:25 2015
        violated 499852 bytes,
          5-min violate rate 782 bytes/sec
          peak rate 867 bytes/sec at Tue Jul 14 03:14:24 2015