Ask the Expert: ASR9000 Hardware Architecture

Unanswered Question
Jan 29th, 2014

With Satya Narra and Xander Thuijs 

Satya NarraXander Thuijs

Welcome to the Cisco Support Community. This is an opportunity to learn and ask questions about software installation, SMU installation, ASR9000 architecture, installing satellite (ASR9000 nv), and troubleshooting of the Cisco ASR 9000 Series Aggregation Services Routers with experts Satya Narra and Xander Thuijs.

The Cisco ASR 9000 Series Aggregation Services Routers product family offers a significant added value compared to the prior generations of carrier Ethernet routing offerings. The Cisco ASR 9000 Series is an operationally simple, future-optimized platform using next-generation hardware and software. 

Satya Narra is a network consulting engineer and specialist in ASR9000 deployments for service provider networks. He is an expert in many technology areas, including IP routing, MPLS, BGP, OSPF, multicast, system architecture, and network design. Satya has a master of science degree in electrical engineering from Wichita State University, Wichita, KS. He started his career as a customer support engineer in TAC/HTTS supporting some of the biggest networks, including Comcast, Citi, Amazon, JPMorgan, AT&T, Vodofone, and others.   


Xander Thuijs is a principal engineer for the Cisco ASR 9000 Series and Cisco IOS-XR product family at Cisco. He is an expert and advisor in many technology areas, including IP routing, WAN, WAN switching, MPLS, multicast, BNG, ISDN, VoIP, Carrier Ethernet, System Architecture, network design and many others. He has more than 20 years of industry experience in carrier Ethernet, carrier routing, and network access technologies. Xander  holds a dual CCIE certification (number 6775) in service provider and voice technologies. He has a master of science degree in electrical engineering from Hogeschool van University in Amsterdam.

Remember to use the rating system to let Satya and Xander know if you have received an adequate response.

Satya might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation in Service Providers community,  sub-community, XR OS and Platforms discussion forum shortly after the event. This event lasts through February 14, 2014. Visit this forum often to view responses to your questions and the questions of other community members.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (3 ratings)
Michel Bou Sleiman Mon, 02/03/2014 - 00:16

Dear Satya,

Kindly I have a few questions:

1- When will the RSP-880 be available?

2- Is the CSC-CE VRFand CSC-PE VRF supported on the same chassis/line card?

3- How to troublehsoot hardware drops?

Thanks a lot for your support.

Best regards,

Michel.

Xander Thuijs Mon, 02/03/2014 - 05:18

Michel,

1- When will the RSP-880 be available?

This is part of the "Tomahawk" (NP5) delivery which is end of this year.

2- Is the CSC-CE VRFand CSC-PE VRF supported on the same chassis/line card?

Yes there wouldnt be any restrictions

3- How to troublehsoot hardware drops?

there are quite a few good documents on that here on the support forums in this xr and os platform section.

for instance: https://supportforums.cisco.com/docs/DOC-15552 you may like.

Also check and download cisco live presentation 2904 from the CL site from Orlando 2013 that you may like with even more detail and fabric stuff in it.

regards

xander

Xander Thuijs CCIE #6775
Principal Engineer 
ASR9000, CRS, NCS6000 & IOS-XR

Michel Bou Sleiman Mon, 02/03/2014 - 06:30

Hi Xander,

Thanks a lot for your reply.

Regarding point 2, just to make one thing clear: what i am looking for is linking two CSC VRFs on the same router (by importing/exporting RTs). Meaning that I have multiple CE sites that need to connect to a central site but over CSC topology. This does not pose a problem i guess when the CE VRF and CENTRAL VRF are on different PE, but I have a situation where the CE VRF will coexist on the PE that holds the CENTRAL VRF.

Is there any limitation to achieve CSC here?

Thanks,

Michel.

Xander Thuijs Mon, 02/03/2014 - 07:08

Hey Michel,

it sounds like you want to import multiple route-targets on the VRF on the CSC-PE side, is that correct?

If so, I dont see any restrictions necessarily.

Although I haven't seen this tested personally, based on the implementation I am not seeig any obstacles either yet.

One question I have though is, howcome the necessity of linking the two vrf's on the PE side, you could do that on the CSC-CE side too, right? then it saves some bw on the PE/CE link in case there is inter vrf traffic.

cheers!

xander

Michel Bou Sleiman Wed, 02/05/2014 - 03:50

Hi Xander,

the setup is as following:

(ISP1_ASBR)----------_____

(ISP2_ASBR)----------_____|----------(CSC_PE_A)----------(INTERNET PE)

                                                          |

(ISP3_ASBR)------------------------------(CSC_PE_B)

We have multiple ISP (ISP1 and ISP2) clients connected to the asme CSC_PE_A. Each ISP is in a unique VRF. as well on CSC_PE_A we have the INTERNET_PE that is part of the INTERNET VRF.

The setup i am looking for is to allow the loopback of the ISPx PE and INTERNET_PE to be transported over the CSC_PE_X, in order to establish multihop EBGP and advertise the required prefixes between ISP_X_ASBR and INTERNET_PE only.... these prefixes must not be installed in any of the CSC_PE.

Best regards,

Michel.

Xander Thuijs Wed, 02/05/2014 - 05:06

that should be possible Michel, the only hmm moment I have is when you mentioned that the customer networks should

not be in the CSC PE... I am not sure how that is going to work then...

Here are some visualizations as to how it would come together:

Screen Shot 2014-02-05 at 8.01.08 AM.png

Screen Shot 2014-02-05 at 8.01.22 AM.png

Screen Shot 2014-02-05 at 8.01.34 AM.png

tomek0001 Mon, 02/03/2014 - 10:41

Where can I find more information the the RSP-880? What are the majore enhancements and features that it will provide?

Thank you.

Xander Thuijs Mon, 02/03/2014 - 13:20

Hi Tomek,

I dont have colleteral ready for the new RSP and linecards yet.

The new RSP merely has a faster CPU (of course), and an updated fabric to support the new higher density linecards

(eg 48x10, 6x100G) which are then Tomahawk based (NP5).

The features are mainly driven by the ucode of the NP. and these linecards will have more room for more features and of course faster speeds (in terms of BW and PPS).

I think that by spring/summer this year we should be able to post some stuff out for the new cards with more details as we are still working on a lot of things for the launch later/nearing the end of this year.

regards

xander

Satya Narra Mon, 02/03/2014 - 15:21

Thanks Xander

Lisa Whitle Wed, 02/05/2014 - 16:08

Hi Satya,

Thank you for this helpful discussion.  Could you please explain MC-LAG Switchover types? Appreciate your help on this.

Thank you.

Lisa

Satya Narra Wed, 02/05/2014 - 16:55

Hi Lisa,

The mLACP switchover method can be modified using the following CLI:

RP/0/0/CPU0:IOSXR(config)#interface bundle-ether 1 mlacp switchover type ?
  brute-force    Force switchover by disabling all local member links
  revertive      Revert based on configured priority values
RP/0/0/CPU0:ios(config)#interface bundle-ether 1 mlacp switchover type 

These options(and the default, when this configuration is not present) are mutually exclusive, and the configuration must match on the bundle on both POA's. They determine the dynamic priority manamgement or brute force mechanism is used, and whether the behaviour is revertive or non-revertive.

Dynamic Priority Management

Switchover is acheved gracefully by modifying the LACP port priorites.

Brute force

This does not involve any priority changes. When using brute force, the operational priority always matches the configured priority. A Switchover is performed when the bundle goes down on the active POA. A switchover is also performed, If active-minimum-links is not satisfied, it brings down other active links on this bundle to force a switchover completely to other PoA.

Revertive

You can configure the MC-LAG in Revertive switchover mode. It means, whenever higher priority links are available, it forces an automatic switchover to that PoA.

Non-revertive

By default switchover is non-revertive. It means,  bundle does not fail back to the original active PoA except when a subsequent failure is encountered. In this mode, POA's do not consider to be primary and secondary, only active or standby. The configured priorities are only used to determine which POA is initally active.

Regards,

--SN

Xander Thuijs Thu, 02/06/2014 - 04:22

I have some nice detail about MC-LAG here too in case you're interested in that topic:

https://supportforums.cisco.com/docs/DOC-34856

In addition, with asr9k nvEdge (aka cluster) you can link 2 separate devices as one logical entity, this means

that you dont need MC-LAG but can use an active active bundle with members on each rack/separate device and eliminate the complex technology of MCLAG and its needs. the two racks dont need to be in the physical location.

If you are interested in the cluster concept here is something on that:

https://supportforums.cisco.com/docs/DOC-34114

regards

xander

Xander Thuijs CCIE #6775
Principal Engineer 
ASR9000, CRS, NCS6000 & IOS-XR

mohsinsohail Thu, 02/06/2014 - 09:03

Hello,

I was hoping you could point me in the right direction to understand the QoS features in the ASR9K.

My use case is very simple. I have a service Provider environment where I have to deploy Traffic Classification ( based on ACL ) and MArking ( DSCP 46 for our VOIP traffic and DSCP 0 for eveything else ) on the Edge ( Our Edge is comprised of 3 seperate locations two of them have ASR9001 and 1 has a 9010 ( in pairs ). The Core is then comprised of various ASR9K variants as well. We have Low Queue Cards.

I am fairly well aquainted with the MQC and know how to create the appropriatte Class-maps, Traffic Policy and apply to interfaces. However, I am not sure about any HW/Architectural  limitations on the ASR9K and furthermore, I dont know the Queueing structure ( for e.g. how could I enable low latency Queuing for DSCP 46 )

Any pointers will be greatly appreicatted. Most of my knowledge is sourced from :

http://www.cisco.com/en/US/partner/docs/routers/asr9000/software/asr9k_r5.1/qos/configuration/guide/b_qos_cg51xasr_chapter_0101.html#ID691

Mohsin Sohail

Thank you

Xander Thuijs Thu, 02/06/2014 - 09:27

Hi Mohsin,

this article https://supportforums.cisco.com/docs/DOC-15592 probably provides for a lot of insight and detail already.

If you need more (and who doesn't ) pull cisco live 2013 presentation ID 2904 with a whole section about QOS.

cheers!

xander

Xander Thuijs CCIE #6775
Principal Engineer 
ASR9000, CRS, NCS6000 & IOS-XR

mohsinsohail Fri, 02/07/2014 - 06:28

Thank You Kindly, Looks like I have tonnes to read now

mohsinsohail Tue, 02/11/2014 - 09:47

AFter reading and listening to quite alot about the QoS Features on the 9K, I have one more question to ask. Is my understanding correct that without a service policy applied to an interface I cant really see the state of the Queues on a Line Card nor can I see the Packet profiles from a DSCP perspective.

Here is the Context: Currently I have no QoS enabled on my 9Ks which means the default trust behavior is occuring on the interface. I want to currently see what raffic is hitting what Queues and what DSCP markings are coming in ( per port,per queue or mayebe even a rate ) - Its kind of validating what I already know is happening( which is packets are just hitting the Normal Queue ). Lets call this a sanity check

Then once I apply my Service Policy ( Classify packets per an ACL, mark per the ACL classification and Place in Queue per markings ) - I would then again want to run the same sanity check to see what happens.

Mohsin

Xander Thuijs Tue, 02/11/2014 - 12:46

Mohsin; without a qos policy applied there really no queue other then the interface default queue. so there is no priority separation at that point.

In order to find out what the distribution is of the different TOS/DSCP's then you'd need to define a qos policy-map that matches all those different values you're interested in and basically do 'nothing' in the policy-map class definition.

That way you can find out from the pmap counters what the different rates per class are.

Note that you dont necessarily need a "queue" in order to do that verification. You only need that classification.

Priority and bandwidth create that queue on egress, bandwidth on ingress creates a queue. Priority on ingress just provides that marking, but doesnt instantitate a queue.

A queue instantiation provides for per class Q'ing in case of congestion, where as a different policed classes will use the same Queue ID and if the interface is congested there is no guarantee per class what makes it through as everything uses that same queue.

regards

xander

DEEPAK ARORA Sat, 02/08/2014 - 07:14

Any plans to make ASR9K, CRS & NCS series a digestable piece as well for for people newly introduced to platform. Now don't get me wrong but there are lot of missing pieces when it comes to knowing these platforms.

> No Books.... which is perhaps always the best way to start with new things

> Self Placed Learning Material ...Maybe PEC

Hard to understand the piece like same line card in two flavors as transport or service

or maybe what should be criterias to help me decide among CRS Vs ASR9K Vs NCS

What are different features these platform support whcih regular or old ones like 7600 etc dont

How redundancy mechasims works within these platoforms, QOS etc

Here is good example about what I am trying to say:

http://www.ine.com/all-access-pass/training/playlist/ccie-data-center-nexus/nexus-hardware-architecture-100121011.html

I can find ASR material in bits and pieces from Cisco.com or Cisco Live. But As a Engineer I prefer single point. Also Cisco DOC Wiki might be another place where you may want all things to put together.

Make it friendly and it will bring more friends . Don't make it look like Rocket Science

HTH...

Deepak Arora

Evil CCIE

http://deepakarora1984.blogspot.com

Xander Thuijs Sat, 02/08/2014 - 08:33

hi deepak, that is great feedback.

I totally recognize the need for documentation and learning capabilities. And there are things in motion for that big time.

The book portion is interesting you note, because when we wanted to publish a comprehensive book about XR the publisher declined that because based on their "data" there was no interest. So I guess you proved them wrong .

Anyways, a book may not happen the way I had hoped and I guess you had wished for either. So my strategy is to leverage the XR OS support forum, blogs, documents and Q&A as the vehicle to get as much information out there as possible in bite size pieces pertaining to a particular technology (area) or design.

In the last year or two we have put a tremendous amount of technotes out there based on what we are learning from suggestions received like yourself.

Because many of the folks contributing are engineers whose primary focus is not documentation, I can't just have them write stuff for the sake of writing, so I need that direction in order to understand where the different topics of need lie.

Coming soon the support forums will get an overhaul into a new style based on open source technology which likely provide for more interaction capabilities then what is there today.

As for learning capabilities; soon we'll have XR-VR's avaiable to build and test configurations with on soft routers

which provides for a great deal of learning already.

For the technical pieces; I'd like to have some direction and a few topics for you that you are missing so we can get that piece out there (via the support forums).

thanks!!

xander

Xander Thuijs CCIE #6775
Principal Engineer 
ASR9000, CRS, NCS6000 & IOS-XR

feene1@hotmail.com Sun, 02/09/2014 - 19:27

Hello,

With IOS-XR 5.1.1 on the 9K I see that there is a QOS offload feature.

http://www.cisco.com/en/US/docs/routers/asr9000/software/asr9k_r5.1/qos/configuration/guide/b_qos_cg51xasr_chapter_0101.pdf

I think I am reading this right - are we now able to shape outbound on a client 9000V port and not use a queue on a TR card?  Essentially "offload"?

Any additional insight to what this features provides would be much appreciated. 

Xander Thuijs Mon, 02/10/2014 - 04:28

hi Feene1,

the qos offload helps with the satellite fabric extension connection oversubscription.

If the ICL (inter chassis link) is oversubscribed for whatever reason, then the qos offload will help

on ingress policing more or less on the sat ports to prevent the oversubscription to happen.

cheers

xander

tenaro.gusatu.novici Mon, 02/10/2014 - 06:32

Hi there,

I'm new in this but hopefully you can give me quick hints why/when ASR9k is better choice than CRS-X or NCS6K series?

Thanks,

Tenaro

Xander Thuijs Mon, 02/10/2014 - 06:42

hi Tenerao,

it somewhat depends on where the device fits in the network; If you are looking at the edge with a need for edge like features, then I would say, you probably are looking at an ASR9000.

If you need a core device without an extensive need for flexibility and features, then probably the CRS/NCS6k is what you want to be focussing on.

There is overlap between the 3 products you mention, which is awesome because it makes them more versatile, but I can also see the confusion as to what to choose when.

Without having a lot of background about your network (needs), it is hard to provide a crisp and sound answer.

I would recommend to consult with your account team and have a Technical Marketing Engineer and a Product Manager piped in to give you the best advice for strategy on this one.

regards

xander

Mark Pace Balzan Tue, 02/11/2014 - 11:33

Hi,

Ive followed the cisco live presentations which are great to explaining the architecture of the ASR9k.

I'd like to ask few questions to ensure Im getting things correctly:

Q1. I understand that with dual RSP440 in a system, we have total fabric capacity of 220G in / 220G out between each LC and each RSP, so total aggregate of 880G assuming both RSP are available.  For the typhoon 24x10GE card, we have 240G in / 240G out from the LC, so this is fine with dual RSP, but if we have one RSP, then we have very slight oversubscription right ?  Probably this is negligible in practical use as its unlikely all interfaces will simultaneosuly running at full rate but just want to ensure im getting this right.

Q2. For the MOD160 card, is the total capacity of this card 160G making it a rather low capacity card compared to the 24 and 36x10GE ?   Im not sure I follow since this is far less than even a single RSP can handle but seems to be the newer cards launched ?

Q3. Specifically for the ASR9001, which of the fixed 10GE onboard ports are mapped to which NPU internally and with which bay ?  Would it make sense to have 'core uplinks' from the fixed 10G ports coming from separate NPUs ?

thanks

Mark

Xander Thuijs Tue, 02/11/2014 - 12:41

Mark:

Q1) Correct, on single RSP, you are missing 20G, the way that it works is each of the 4 FIA's gets 5G back pressure to maintain fairness. So if you dont need the 24 interfaces immediately, it is best to wire them 0-4 then 6-9 etc

so you skip one interface per 2 NPU's so you dont run into any issues on a single RSP.

Q2) It is effectively the same as a MOD-80, which only has 1 NPU per bay, whereas the MOD-160 has 2 NPU's per bay. This MOD160 is useful when you want to use the 2x40G MPA, 8x10G MPA or when pps is a concern to you.

with the 2 NPU's per bay this card will provide top notch pps performance as each NP serves half the interfaces of the bay, which is overkil for most MPA's if you ask me .

Q3) 2 onbard 10G's of the 9001 are mapped to a single NPU. So each NPU on the 9001 serves 2x10G Onboard and an MPA bay.

regards

xander

Xander Thuijs CCIE #6775
Principal Engineer 
ASR9000, CRS, NCS6000 & IOS-XR

Mark Pace Balzan Thu, 02/13/2014 - 00:26

Hi xander,

thanks for the fast and accurate reply

so for the asr9001, do we have fixed SFP+ 0,1 and MPA bay0 on one NPU, and SFP+ 2,3 and MPA bay1 on the other NPU, which is possibly also why the 9001S has one NPU and associated ports disabled ?

thanks

Mark

smilstea Thu, 02/13/2014 - 06:09

Hi Mark,

Correct.

Here is a quick way to check for any LC on any ASR9K. Below I have a 20x1GE MPA and 4x10GE MPA installed in my 9001.

RP/0/RSP0/CPU0:ASR9001-A#show controller np ports all loc 0/0/CPU0

Thu Feb 13 09:07:03.220 EAST

                Node: 0/0/CPU0:

----------------------------------------------------------------

NP Bridge Fia                       Ports

-- ------ --- ---------------------------------------------------

0  0      0   GigabitEthernet0/0/0/0 - GigabitEthernet0/0/0/19

0  0      0   TenGigE0/0/2/0, TenGigE0/0/2/1

1  1      1   TenGigE0/0/1/0, TenGigE0/0/1/1, TenGigE0/0/1/2, TenGigE0/0/1/3

1  1      1   TenGigE0/0/2/2, TenGigE0/0/2/3

Thanks,

Sam

daniel.dib Mon, 02/24/2014 - 21:05

Hi Satya and Xander,

I have some questions on ASR9k platform. First some questions on NV satellite:

Is there a list of features not supported on the host? Such as LACP.

Do you have a roadmap for these features?

ASR9000v can't act as transit for traffic, correct?

And for the ASR9k:

Do you support IGMP snooping and PIM snooping under VPLS?

Are there counters available via SNMP for amount of traffic in an EVC?

Is it possible to run VPLS aware Netflow to graph traffic within VFI?

Daniel Dib
CCIE #37149

Please rate helpful posts.

Satya Narra Tue, 02/25/2014 - 07:14

Hello Daniel,

Yes, you are right, ASR9000v cannot act as a transit for traffic

Going forward Satellite will be supporting

With 5.2.x and 5.3.x we will be adding the following features.

Multicast Offload

Bundle over Bundle

Dual head-ARP, ND Sync.

Satellites, use ICCP so there might not be any plans to implement LACP in the future.

Regards,

--Satya Narra

daniel.dib Wed, 02/26/2014 - 10:42

So if you want to dual home a host that is not possible or will the satellites act as one device?

Daniel Dib
CCIE #37149

Please rate helpful posts.

echami Wed, 02/26/2014 - 19:06

You can dual host in 5.1.1. You will need a sat lic on each 9k that is hosting that Sat.
Check 511 asr9000 release notes for more details.
Eddie.

Sent from Cisco Technical Support iPhone App

daniel.dib Wed, 02/26/2014 - 21:31

What I meant was the end host (customer) not the NV host.

Daniel Dib
CCIE #37149

Please rate helpful posts.

Actions

Login or Register to take actions

This Discussion

Posted January 29, 2014 at 5:05 PM
Updated February 14, 2014 at 1:21 PM
Stats:

Related Content