- Cisco Community
- Technology and Support
- Data Center and Cloud
- Data Center and Cloud Knowledge Base
- Packet Loss When Pinging From/To A Nexus 7000
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
07-05-2012 02:17 PM - edited 03-01-2019 05:56 AM
Introduction
Cisco Nexus 7000 Series Switches offer one of the most comprehensive data center network feature sets in a single platform. As ping is a common tool used to test connectivity in networks, it is not uncommon for users to try to ping a Nexus 7000 as a test. However, when pinging a Nexus 7000, it is very common to see packet loss due to the default behavior in which the Nexus 7000 uses Control Plane Policing (CoPP) to rate-limit certain types of traffic to the CPU.
The main asset of the Nexus 7000 (as well as other Cisco switches) is the ability to switch packets in hardware using high speed ASICs. The CPU however, is not designed for extensive packet forwarding and is typically reserved for the processing and creation of control plane packets such as routing protocol hellos, BPDUs, LACP, CDP, etc. Loss of these control plane packets can create network instability, so it is very important to protect the CPU from traffic that could potentially cause high utilization. This is the main motivation behind CoPP on this platform.
NOTE - CoPP only applies to traffic destined to the switch itself (control plane). It has no impact on hardware switched traffic through the box.
CoPP
CoPP is a hardware-based feature that protects the Supervisor from DoS attacks. It controls the rate at which packets are allowed to reach the Supervisor. The CoPP feature is modeled like an input QoS policy attached to the special interface called the control-plane. However, CoPP is a security feature and not part of QoS. In order to protect the Supervisor, the CoPP separates data plane packets from the control plane packets. After a packet is classified, the packet can also be marked and used to assign different priorities based on the type of packets. Conform, exceed, and violate actions (transmit, drop, mark-down) can be set. If no policer is attached to a class, then a default policer is added whose conform action is drop. Glean packets are policed with default-class. One rate, two color, and two rate, three color policing are supported.
Example
Here is an example of the type of packet loss one can expect when trying to ping a Nexus 7000. The ip address 10.48.1.50 is an ip address of an SVI configured on a Nexus 7000 :
StaticVSS#ping 10.48.1.50 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 10.48.1.50, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (994/1000), round-trip min/avg/max = 1/1/4 ms
StaticVSS#
A quick Ethanalyrzer on the N7K shows these ICMP requests/replies at the CPU level of the N7K, therefore making it prone to rate-limiting by CoPP.
Nexus_7000# ethanalyzer local interface inband capture-filter "host 10.48.1.50 and icmp" limit-captured-frames 100 | no-more
<snippet>
Capturing on inband
2012-07-04 16:28:45.723631 10.48.1.49 -> 10.48.1.50 ICMP Echo (ping) request
2012-07-04 16:28:45.723649 10.48.1.50 -> 10.48.1.49 ICMP Echo (ping) reply
2012-07-04 16:28:45.724063 10.48.1.49 -> 10.48.1.50 ICMP Echo (ping) request
2012-07-04 16:28:45.724200 10.48.1.50 -> 10.48.1.49 ICMP Echo (ping) reply
2012-07-04 16:28:45.724562 10.48.1.49 -> 10.48.1.50 ICMP Echo (ping) request
2012-07-04 16:28:45.724694 10.48.1.50 -> 10.48.1.49 ICMP Echo (ping) reply
2012-07-04 16:28:45.725062 10.48.1.49 -> 10.48.1.50 ICMP Echo (ping) request
2012-07-04 16:28:45.725192 10.48.1.50 -> 10.48.1.49 ICMP Echo (ping) reply
2012-07-04 16:28:45.725561 10.48.1.49 -> 10.48.1.50 ICMP Echo (ping) request
2012-07-04 16:28:45.725692 10.48.1.50 -> 10.48.1.49 ICMP Echo (ping) reply
First we check to make sure there is a CoPP policy applied with the command "Show copp status". To view the policy, you can use “show policy-map interface control-plane”.
Nexus_7000# show copp status
Last Config Operation: None
Last Config Operation Timestamp: None
Last Config Operation Status: None
Policy-map attached to the control-plane: copp-system-p-policy-strict
Here is an example of the global config showing a policy appled under the control-plane:
control-plane
service-policy input copp-system-policy
Looking further at the N7K, we can see that there is a class for ICMP that matches an ACL with ACEs referencing echo request/replies..
Nexus_7000# show access-lists copp-system-acl-icmp
IP access list copp-system-acl-icmp
10 permit icmp any any echo
20 permit icmp any any echo-reply
Nexus_7000# show class-map type control-plane copp-system-class-monitoring
class-map type control-plane match-any copp-system-class-monitoring
match access-group name copp-system-acl-icmp
match access-group name copp-system-acl-icmp6
match access-group name copp-system-acl-traceroute
Here we confrim what we suspected above --- ICMP is being rate-limited and dropped at the CPU. By default, ICMP traffic will be policed with a cir of 130 kbp per slot. NOTE - In the below example module 3 is the module where the traffic is ingressing/egressing
Nexus_7000# show policy-map interface control-plane class copp-system-class-monitoring
Control Plane
service-policy input: copp-system-policy
class-map copp-system-class-monitoring (match-any)
match access-group name copp-system-acl-icmp
match access-group name copp-system-acl-icmp6
match access-group name copp-system-acl-traceroute
set cos 1
police cir 130 kbps , bc 1000 ms
module 1 :
conformed 0 bytes; action: transmit
violated 0 bytes; action: drop
module 3 :
conformed 543980 bytes; action: transmit
violated 3540 bytes; action: drop
module 4 :
conformed 0 bytes; action: transmit
violated 0 bytes; action: drop
FAQ
Q – How did the CoPP configuration get added? I don’t recall ever explicitly adding it.
A- CoPP is typically added during the initial setup configuration of a vDC, so this is likely when the configuration was generated.
Q – I am considering removing CoPP. Is there any downside in doing so?
A – Cisco recommends keeping CoPP in place in order to provide a level of protection at the control-plane level. However, it may be necessary to tweak the CoPP config in some cases depending on your specific traffic requirements.
Q – I have other Cisco switches in my network. Why is CoPP not enabled on those platforms?
A - Although other Cisco switches like the 6500 and 4500 support CoPP, only NX-OS based switches have it enabled by default. More information about CoPP on a particular platform can be found in the supporting configuration guides.
Related Information
Cisco NX-OS and Virtual Device Contexts (VDCs)
VRF Configuration and Verification on Nexus 7000
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:
