With Gilles Dufour
Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about how to configure and troubleshoot Cisco 3GPP Mobility solutions, specifically the Cisco ASR 5000 with expert Gilles Dufour.
The Cisco ASR 5000 Series combines massive performance and scale with flexibility, virtualization, and intelligence, so network resources are available exactly when they are needed. The series was developed to address the anticipated increase in performance requirements that the next generation of the mobile Internet will bring. Join expert Gilles Dufour as he answers your questions about configuring and troubleshooting the Cisco ASR 5000 Series.
Gilles Dufour is a technical leader in the Mobility Business Unit. Before joining the Mobility group, Gilles was part of the data center team in charge of all Cisco load balancers (CSM, CSS, ACE). Gilles has more than 15 years of experience inside Cisco. During his career, Gilles achieved his CCIE in routing and switching (1998) and security (2002) (CCIE 3878).
Remember to use the rating system to let Gilles know if you've received an adequate response.
Because of the volume expected during this event, Gilles might not be able to answer every question. Remember that you can continue the conversation in the Wireless - Mobility community, subcommunity, Security and Network Management, shortly after the event. This event lasts through November 29, 2013. Visit this forum often to view responses to your questions and those of other Cisco Support Community members.
I have many questions and will start with troubleshooting one: what are the best commands to capture incoming and/or outgoing messages for particular network element? To make it more realistic, let's say we are focused on MME and I'm trying to figure out if anything is coming from S1 interface (one case) or if MME is sending anything towards HSS (second case).
Thanks for this discussion!
Thanks for the first question.
In terms of troubleshooting, typically I will first identify the service involved. So taking your example of MME device, I would look at the mme-service.
You can then check with 'show mme-service statistics' what is happening.
You can use the same command and provide extra argument like 's1ap' to limit the statistics to the S1 interface.
Here is an example from my lab
[mme]xt2-4# show mme-service statistics s1ap verbose | more
Transmitted S1AP Data:
S1 Setup Resp: 1 S1 Setup Fail: 0
Reset: 0 Reset Ack: 0
Overload Start: 0 Overload Stop: 0
MME Dir Info Transfer: 0 Paging: 0
eNB Config Update Ack: 0 eNB Config Update Fail: 0
S1AP Msg Encode Fail: 0 E-RAB Setup Req: 4
E-RAB Modify Req: 0 E-RAB Release Command: 0
Initial Ctxt Setup Req: 4 UE Ctxt Release Command: 39
UE Context Modify Req: 0 Downlink NAS Transport: 95
Error Ind: 0 Handover Command: 0
Handover Prep Fail: 0 Handover Request: 0
Handover Cancel Ack: 0 Path Switch Request Ack: 0
Path Switch Req Fail: 0 Downlink S1 CDMA2000: 0
Trace Start: 0 Deactivate Trace: 0
MME Status Transfer: 0 Loc Report Control: 0
MME Config Update: 0 S1AP Encode Fail: 0
MME Config Transfer: 0 Kill Request: 0
Downlink Non-UE LPPaTpt: 0 Downlink UE LPPaTpt: 0
Received S1AP Data:
S1 Setup Req: 1 Reset: 0
Reset Ack: 0 eNB Dir Info Transfer: 0
eNB Config Update: 0 S1AP Msg Decode Failure: 0
S1AP Msg Unexpected: 0 E-RAB Setup Resp: 4
E-RAB Modify Resp: 0 E-RAB Release Resp: 0
E-RAB Release Ind: 0 Initial Ctxt Setup Resp: 4
Initial Ctxt Setup Fail: 0 UE Context Release Req: 0
UE Ctxt Release Comp: 39 UE Context Modify Resp: 0
UE Ctxt Modify Fail: 0 Initial UE Message: 40
Uplink NAS Transport: 71 NAS Non-Delivery Ind: 0
Error Indication: 0 Handover Request Ack: 0
Handover Cancel: 0 Handover Required: 0
Handover Fail: 0 Handover Notify: 0
Path Switch Req: 0 eNB Status Transfer: 0
UE Capability Info Ind: 0 Uplink S1 CDMA2000: 0
Trace Failure Ind: 0 Location Report: 0
Loc Report Fail Ind: 0 S1AP Decode Fail: 0
MME Config Update Fail: 0 MME Config Update Ack: 0
S1AP Unexpected Event: 0 eNB Config Transfer: 0
Uplink Non-UE LPPaTpt: 0 Uplink UE LPPaTpt: 0
Kill Response: 0
Similarly, you can execute "show hss-peer-service statistics all".
You can also check on the protocol side for statistics.
Like for diameter connections, you can use "show diameter statistics endpoint MME-S6A" where in my case my endpoint name is MME-S6A
So just to confirm, there is nothing similar to debug commands in IOS where you can see in the real time what is happening?
Yes, we do have debugging like feature in Staros.
It's called 'monitor subscriber' and 'monitor protocol'.
For example if you want to see all activities related to a certain UE, you do a 'monitor subscriber msid .....'
[local]xt2-2# monitor subscriber msid 111222600100000
Multiple Matching Calls (2) - Monitoring (1) call(s):
Matching Call Found:
MSID/IMSI : 111222600100000 Callid : 017dc661
IMEI : n/a MSISDN : 148985149513848
Username : 148985149513848 SessionType : sgw-pdn-type-ipv4
Status : Active Service Name: sgw-svc
Src Context : sgw Dest Context: sgw
C - Control Events (ON ) 11 - PPP (ON ) 21 - L2TP (ON )
D - Data Events (ON ) 12 - A11 (ON ) 22 - L2TPMGR (OFF)
E - EventID Info (ON ) 13 - RADIUS Auth (ON ) 23 - L2TP Data (OFF)
I - Inbound Events (ON ) 14 - RADIUS Acct (ON ) 24 - GTPC (ON )
O - Outbound Events (ON ) 15 - Mobile IPv4 (ON )
S - Sender Info (OFF) 16 - A11MGR (OFF) 26 - GTPU (OFF)
T - Timestamps (ON ) 17 - SESSMGR (ON ) 27 - GTPP (ON )
X - PDU Hexdump (OFF) 18 - A10 (OFF) 28 - DHCP (ON )
A - PDU Hex/Ascii (OFF) 19 - User L3 (OFF) 29 - CDR (ON )
+/- Verbosity Level ( 1) 31 - Radius COA (ON ) 30 - DHCPV6 (ON )
L - Limit Context (OFF) 32 - MIP Tunnel (ON ) 53 - SCCP (OFF)
M - Match Newcalls (ON ) 33 - L3 Tunnel (OFF) 54 - TCAP (OFF)
R - RADIUS Dict: (no-override) 34 - CSS Data (OFF) 55 - MAP (ON )
G - GTPP Dict: (no-override) 35 - CSS Signal (OFF) 56 - RANAP (OFF)
Y - Multi-Call Trace (OFF) 36 - EC Diameter (ON ) 57 - GMM (ON )
H - Display ethernet (OFF) 37 - SIP (IMS) (OFF) 58 - GPRS-NS (OFF)
40 - IPSec IKEv2 (OFF) 59 - BSSGP (OFF)
41 - IPSG RADIUS (ON ) 60 - CAP (ON )
42 - ROHC (OFF) 64 - LLC (OFF)
43 - WiMAX R6 (ON ) 65 - SNDCP (OFF)
44 - WiMAX Data (OFF) 66 - BSSAP+ (OFF)
45 - SRP (OFF) 67 - SMS (OFF)
46 - BCMCS SERV AUTH(OFF)68 - PHS Control(ON )
47 - RSVP (ON ) 69 - PHS Data (OFF)
48 - Mobile IPv6 (ON ) 76 - PHS EAPOL (ON )
49 - ASNGWMGR (OFF) 77 - ICAP (ON )
50 - STUN (IMS) (OFF) 78 - Micro-Tunnel(ON )
51 - SCTP (OFF)
72 - HNBAP (ON ) 79 - ALCAP (ON )
73 - RUA (ON ) 80 - SSL (ON )
74 - EGTPC (ON )
75 - App Specific Diameter (OFF)
81 - S1-AP (ON ) 82 - NAS (ON )
83 - LDAP (ON ) 84 - SGS (ON )
85 - AAL2 (ON )
86 - PHS(Payload Header Suppression) (OFF)
87 - PPPOE (ON )
88 - RTP(IMS) (OFF) 89 - RTCP(IMS) (OFF)
91 - NPDB(IMS) (OFF)
92 - SABP (ON )
Then you activate the protocols/functions you want to monitor and you will start seeing live what happens for this particular caller.
You can do the same for a protocol but be careful it is CPU intensive.
[local]xt2-2# monitor protocol
MONITOR GLOBAL PROTOCOLS:
11 - SNMP 21 - L2TP (Admin only)
12 - RADIUS Authentication (Admin only) 22 - L2TPMGR (Admin only)
13 - RADIUS Accounting (Admin only) 23 - L2TP Data (Admin only)
14 - A11 (R-P Interface) (Admin only) 24 - GTPC (Admin only)
15 - Mobile IPv4 (Admin only)
16 - A11MGR (Admin only) 26 - GTPU (Admin only)
17 - PPP (Admin only) 27 - GTPP (Admin only)
18 - A10 (Admin only) 28 - DHCP (Admin only)
19 - User L3 (Admin only) 29 - CDR (Admin only)
20 - USERTCP STACK (Admin only) 30 - DHCPV6 (Admin only)
31 - RADIUS COA (Admin only) 51 - SCTP (Admin only)
32 - MIP Tunnel (Admin only) 52 - M3UA (Admin only)
33 - L3 Tunnel (Admin only) 53 - SCCP (Admin only)
34 - CSS Data (Admin only) 54 - TCAP (Admin only)
35 - CSS Signaling (Admin only) 55 - MAP (Admin only)
36 - EC Diameter (Admin only) 56 - RANAP (Admin only)
37 - SIP (IMS) (Admin only) 57 - GMM (Admin only)
38 - IPSec IKE Inter-Node (Admin only) 58 - GPRS-NS (Admin only)
59 - BSSGP (Admin only)
40 - IPSec IKE Subscriber (Admin only) 60 - CAP (Admin only)
41 - IPSG RADIUS Signal (Admin only) 61 - SSCOP (Admin only)
42 - ROHC (Admin only) 62 - SSCFNNI (Admin only)
43 - WiMAX R6 (Admin only) 63 - MTP3 (Admin only)
44 - WiMAX Data (Admin only) 64 - LLC (Admin only)
45 - SRP (Admin only) 65 - SNDCP (Admin only)
46 - BCMCS SERV AUTH (Admin only) 66 - BSSAP+ (Admin only)
47 - RSVP (Admin only) 67 - SMS (Admin only)
48 - Mobile IPv6 (Admin only) 68 - PHS-Control (Admin only)
49 - ASNGWMGR (Admin only) 69 - PHS-Data (Admin only)
50 - STUN (Admin only) 70 - DNS Client (Admin only)
71 - MTP2 (Admin only) 76 - PHS-EAPOL (Admin only)
72 - HNBAP (Admin only) 77 - ICAP (Admin only)
73 - RUA (Admin only) 78 - Micro-Tunnel (Admin only)
74 - EGTPC (Admin only) 79 - ALCAP (Admin only)
75 - App Specific Diameter (Admin only) 80 - SSL (Admin only)
81 - S1-AP (Admin only) 82 - NAS (Admin only)
83 - LDAP (Admin only) 84 - SGS (Admin only)
85 - AAL2 (Admin only)
86 - PHS(Payload Header Suppression) (Admin Only)
87 - PPPOE (Admin only)
88 - RTP(IMS) (Admin only) 89 - RTCP(IMS) (Admin only)
90 - LMI (Admin only) 91 - NPDB(IMS) (Admin only)
92 - SABP (Admin only)
(B)egin Protocol Decoding, (Q)uit,
Allow me to conclude this mini thread with one final question: is there a way to capture full traffic on particular port or interface and save it in a file so it can be opened/analyzed in Wireshark?
There is hidden commands to access a linux shell of every card inside the chassis.
From the linux shell you can use tcpdump to collect a sniffer trace of interfaces.
However, it is not a standard operation and not recommended to do unless instructed by Cisco support.
We prefer to first use the 'monitor subscriber' or 'monitor protocol' command mentioned above.
With those commands, you can activate the hexdump option, and with a small script you can concatenate the hexdumps into a pcap file.
Me again with another one: I'm not sure about purpose of X2 interface and would like to know if it is possible to connect data stream between two UEs without anchoring it on SGW (UE1 - eNodeB1 - eNodeB2 - UE2)?
As far as I know you can't bypas SGW for inter UE communication using X2 interface.
The main purpose of X2 interface is for UE handover between enodeb.
Thanks for your answers! I'm not sure if I really understand benefit(s) of X2 interface and will appreciate if you can explain it in little bit more details. As far as I see it, things will work just fine even without X2, right?
Intra E-UTRAN Handover is used to hand over a UE from a source eNodeB to a target eNodeB using X2 when the MME is unchanged.
This is explained in a very good way at the following location :
So basically, the X2 interface helps offload some of the HO task from MME.
I hope this helps.
Evan, when you have redundancy, not just on ASR5k, you typically have 2 options.
One is active-active and the other one is active-standby.
So with 'mode loadshare' both peers are active while with standby more one peer is active and the other one is standby so basically just waiting for the primary to fail to become active.
Could you please tell me how does the GGSN get the username field that we see in monitor subscriber ? Thanks in advance for your help.
I'm not sure what additional value is generated with the following commands:
associate sgw-service mysgwsvc
associate pgw-service mypgwsvc
So the question is: what is lost if upper section is not configured? I simply don't see what is the benefit of configuring these three extra lines.
SAEGW is a combination of SGW/PGW services, once you add following three lines SGW/PGW services (in same ingress context) get associated with each other.
If a subscriber lands on SGW/PGW service part of SAEGW, then we collapse two call lines into single call line. This basically saves one call line across chassis, effectively doubling the session handling capacity of the system.
I'm interested in the WiFi offload feature: what elements are used, apart from LTE on one side and WiFi network on another, to enable seamless handover? Please focus only on mandatory elements, i.e. what is the minimum required in the lab during the first phase (just to make it work).
Assuming you have LTE environment and Wifi network, what you need is a gateway between the 2.
The ASR5k configured as an HNbGW can service this purpose.
Some example here
Or the ASR1k setup as an ISG.
See slide 42 from
And what wil be the purpose of WAG (wifi access gateway)? Is it running as another service on existing StarOS (one that is capable of running SGW or PGW or MME service)?
Indeed the ASR5k could also be configured with eWAG service to interconnect Wifi world with LTE.
It all depends on the wifi you want to use.
Maybe it is clearer from this link
But basically, with an ASR5k you're sure to have the right equipment to interconnect both world.
You may have to play a bit with the config to find the right setup for your environment.
I find the following document also very interesting
Here is a potential solution
a) “rau-inter restrict access-type umts/gprs all” To restrict Inter SGSN handoff
b) ”rau-inter access-type umts/gprs all failure-code 10” To set the gmm cause “Implicit Detach(10)” in RAU Reject msg.
Could you please help me understand the purpose of PCRF and OCS elements: I'm trying to figure out why the functionality of PCRF/OCS elements is not built inside HSS, i.e. what is the benefit of having separate elements? In addition, when talking about basic Attach procedure, can we skip AAA, PCRF and OCS elements and keep only EPC plus HSS? If it is true that AAA, PCRF and OCS are not mandatory elements, what messages/parameters are used to define when each of them will be included?
Thanks for the great work and fantastic topic, we hope to see you here again!