CRS-1 and CRS-3 short recap
The reader of the following article should be familiar with MQC and the corresponding CLI on CRS-1 and CRS-3.
The QoS implementation on CRS-X follows the SSE architecture on the LCs. QoS components on CRS-X are not shared between the CRS-1 and CRS-3 implementation. Although the config and CLI is alike. This includes the 3 components of MQC
- class-map
- policy-map
- service-policy
as well as the majority of the debug and "show" commands.
On CRS-1 and CRS-3 QoS can be configured on the components as follows:
- ingressq - service-policy input
Addresses hw queues on ingressq asic. Priority traffic will use the HP queue on the fabric and fabricq asic.
- egressq - service-policy output
Addresses hw queues on egressq asic.
- fabric-QoS - switch-fabric service-policy
Addresses hw queues on fabricq asic. Priority traffic will also use the HP queue on the fabric.
The packet buffer pool on ingressq an egressq is 1GB. This buffer pool is shard by all ports on the corresponding LC.
QoS on CRS-X
Because CRS-X introduces a new architecture there are some differences compared to CRS-1 and CRS-3.
- On CRS-1 and 3 we have two sets of queues on ingressq ASIC, the shape queues and fabric queues.
- On CRS-X we don't have the concept of ingress queuing. Thus, an ingress service-policy does not accept queuing related commands in any class like "bandwidth remaining percent", "random-detect" and "queue-limit".
- On CRS-X ingress the shape queue is removed and we only have fabric queues. There are 6144 Fabric Queues on each MSC/FP-X. 3072 are UC HP, 3072 are UC LP queues. In addition there is 1 Multi-Cast Lo and 1 Multi-Cast Hi priority queue per (Fabric Interface Asic) FIA.
- On the fabric stages S2 and S3 Unicast HP and LP, Multicast HP and LP queues are present as part of the Fabric Switch Element (SFE) ASIC.
When an ingress service-policy is applied containing a "priority" class priority queues on the ingress FIA on S2 and S3 as well as on the destination (egress) FIA are used. In case of multicast traffic it is placed on the corresponding HP multicast queue.
An ingress service-policy can be deployed together with fabric QoS. Please note, fabric QoS will overwrite the priority class match of the ingress service-policy.
Before 5.3.0 there is service impact of 7% on the ingress LC if ingress QoS and fabric QoS is deployed together.
Egressq packet buffer on CRS-X LCs
On CRS-X we have an egress packet buffer pool of 750MB per NPU (PAT). This pool is shared by the ports belonging to the NPU (slice). 30ms @ 200G per NPU is available.
Congestion on CRS-X LCs
Because the discard mechanism is the preferred behavior in case of congestion towards the NPU a discard shaper has been implemented on CRS-X LCs between FIA and NPU.
The discard mechanism causes traffic-drop at source at per-queue granularity instead of penalizing all the traffic coming to a fabricq destination.
Initially the discard shaper was implemeted with 110G. Since 5.1.3 it is 105G. If traffic destined to a e.g. 100GE egress port goes beyond 105G backpressure and discard drops will be observed. The corresponding fabric queue is building up.
Example outputs (110G is sent to the FIA):
RP/0/RP0/CPU0:CRS-X#show controllers fabricq queues instance 0 loc 1/4/cpu0
|HundredGigE1/4/0/3 | 1| 129| 153| BE| 18| 76| 37500| 33750| 27216|100000000|
SNIP
RP/0/RP0/CPU0:CRS-X#sh controllers fabricq stat detail loc 1/4/CPU0
SNIP
(+ 3132319182 ) => 1.7GOn the ingress LC discard drops are incrementing:
RP/0/RP0/CPU0:CRS-X#show controllers ingressq statistics location 1/10/cpU0 | in discard
Wed Dec 7 14:27:51.064 UTC
discard drops : 190978667452 ( 9636990959268 bytes)
RP/0/RP0/CPU0:CRS-X#show controllers ingressq statistics location 1/10/cpU0 | in discard
Wed Dec 7 14:27:51.983 UTC
discard drops : 190978947130 ( 9637005222846 bytes)
If congestion in this direction is expected three options are available to avoid drops of priority traffic. The assumption is an egress service-policy has been configured. It won't cover traffic > 105Gbps.
(1) Fabric QoS
Here we have three queues available, priority, AF and BE. Fabric QoS is enabled globally and the corresponding service-policy applies system wide.
Example:
RP/0/RP0/CPU0:CRS-X(config)#switch-fabric service-
(2) Ingress QoS
Has to be configured on all ingress interfaces.
(3) Ingress QoS AND fabric QoS.
It is important to remember that if an Ingress QoS policy is applied to an interface and the Fabric QoS policy has been to the router, then the Ingress MSC/FP’s RX PSE will be required to perform two classification cycles.
This will have an impact on the forwarding capacity of the linecard. It is about 20% on CRS-3 and CRS-1 based LCs.
Ingress QoS and fabric QoS should be in sync and should complement each other.
