Datacenter troubleshooting guide - day 3

Blog

Nov 22, 2010 1:42 AM
Nov 22nd, 2010

“Datacenter troubleshooting guide” – a blog by  Gilles Dufour.

Day 3 – Inspecing DP info

Last week we had a look into the 'show np 1 access-list trace' command to figure out if our configuration was correctly interpreted by ACE.

Today, we're going to look into the other information we can get from this command.

But before, let's talk about the ACE architecture.


An ACE module or ACE appliance has 2 important components.  The Control Plane (CP) and the Data Plane (DP).

The Control Plane is your administration CPU.  It's the one that controls the console port, telnet/SSH access, probes, files, configuration,...

The Data Plane is the part that does the routing/switching/loadbalancing actions.

When you add a new command to your configuration, a new class-map, or a new policy for example, the CP will have to merge this information with the current configuration.  There is a process called aclmerged responsible for this operation.

switch/Admin# show proc cpu | i acl
  969         5239     36427    143    0.0   0.0 %   0.0 %   0.0 %  aclmerged

aclmerged takes your text configuration and translates/merges it into a structure called a merged-list.

You can see it for each inteface with the command "show acl-merge merged-list vlan 20 in".

The merged-list is then used to programm the Data Plane.

If the aclmerged process failed for some reasons, an incomplete or incorrect merged-list could be sent to the DP.

This is why it is important to use the 'show np 1 access-list trace' command to verify your configuration as seen on the DP.

Let's go back to our command of interest for today.

switch/Admin# show np 1 access-list trace vlan 20 in pro 6 source 192.168.20.45 0 des 192.168.20.122 80

...

action node 0x44a1240
Action Leaf-node
version+aceid 0x408 (version 0 ace_id 1032 dirty no)
action_flag 0x2 (permit yes log no punt_to_cp no capture no bridge no)
path ID 0x0
src nat 0x0 dst nat 0x0 vserver 0x51 fixup 0x0
TCP conn 0x0 AAA 0x0 Websense 0x0 QOS Policer 0x0
Syslog Info 0

We have talked about the vserver 0x51 already.

Now, let's add a source nat statement to our existing configuration

switch/Admin# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
switch/Admin(config)# policy-map multi-match SLB
switch/Admin(config-pmap)# class VIP-122-80
switch/Admin(config-pmap-c)# nat dynamic 1 vlan 40
switch/Admin(config-pmap-c)#

If we check the DP setup, we will see a src nat action.

switch/Admin# show np 1 access-list trace vlan 20 in pro 6 source 192.168.20.45 0 des 192.168.20.122 80
...
action node 0x44a1940
Action Leaf-node
version+aceid 0x440 (version 0 ace_id 1088 dirty no)
action_flag 0x2 (permit yes log no punt_to_cp no capture no bridge no)
path ID 0x0
src nat 0x1b dst nat 0x0 vserver 0x51 fixup 0x0
TCP conn 0x0 AAA 0x0 Websense 0x0 QOS Policer 0x0

Syslog Info 0
Hitcount 0
Syslog info:
  idx:[1088:0] name_idx:[0:0] hash1:0x0 hash2:0x0 name_len:0 invalid

src nat 0x1b ( 27 in decimal)

switch/Admin# show cfgmgr internal table nat | i 27
27      2          0      DATA_VALID,
switch/Admin#

The object does exist but this command does not give us any other information.

A more useful command is

switch/Admin# show np 1 nat src-nat 27 ?
  <0-65535>  Mapped interface ID

But this command requires an output interface id.

Indeed, nating is done using the natpool of an outgoing interface.

So we need to know that interface id.

To get the interface id, we can use the following method :

switch/Admin# show np 1 interface iflookup
First burnt-in MAC: 00:30:f2:75:f3:f1
Last  burnt-in MAC: 00:30
:f2:75:f3:f7
No of burnt-in MACs: 7
Hostid: 2
Shared vlan macs currently in use (offset from 2048): 0-7
Vlan-vmac indexes currently in use: 0-3
Flags:  Valid shared bridged ftstatus ssl-test normalization icmp-guard switch-m
ode ftvlan remove-eth-pad no-of-lifs

Vlan   ifid matchid ctxt primary vvind ftgrp ttl optact df ma_idx   Flags
----   ---- ------- ---- ------- ----- ----- --- ------ -- ------   -----
1      1    1       0    1       1     100   0   2      0  512      1101000000
10     2    2       0    10      0     100   0   2      0  512      1001000000
20     5    5       0    20      2     100   0   2      0  4608     1101010000
30     3    3       0    30      0     100   0   2      0  512      1011000000
40     6    6       0    40      3     100   0   2      0  12800    1101110000
60     9    9       0    60      0     100   0   2      0  8704     1001100000
77     13   13      0    77      0     100   0   2      0  512      1001000100
330    4    4       0    330     0     100   0   2      0  512      1011000000

Vlan 40 has interface id (ifid) 6.


switch/Admin# show np 1 nat src-nat 27 6

        ID:10 mapped_if:6 policy_id:27 ixp_hint:in IXP1 type:DYNAMIC nat_pool_id
:27
                ID:27 PAT:0 ixp_binding:in IXP1
                lower:172.16.40.1 upper:172.16.40.254 Bitmap-ID:70
                Level 1 Bitmap: 0x1
                Level 2 Bitmap:

So we can see DP will use the range of ip addresses 172.16.40.1 - 172.16.40.254 to source nat traffic hitting our rule 0x51 (81).


Now, let's add a parameter-map to our rule to change the default idle timeout.

switch/Admin# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
switch/Admin(config)# policy-map multi-match SLB
switch/Admin(config-pmap)# class VIP-122-80
switch/Admin(config-pmap-c)# connection advanced-options TIMEOUT-IDLE__10_CONN

And looking at the DP configuration we can see a action has been added to our rule :

switch/Admin# show np 1 access-list trace vlan 20 in pro 6 source 192.168.20.45 0 des 192.168.20.122 80

action node 0x44a2040
Action Leaf-node
version+aceid 0x478 (version 0 ace_id 1144 dirty no)
action_flag 0x2 (permit yes log no punt_to_cp no capture no bridge no)
path ID 0x0
src nat 0x1b dst nat 0x0 vserver 0x51 fixup 0x0
TCP conn 0x51 AAA 0x0 Websense 0x0 QOS Policer 0x0
Syslog Info 0
Hitcount 0
Syslog info:
  idx:[1144:0] name_idx:[0:0] hash1:0x0 hash2:0x0 name_len:0 invalid

TCP conn 0x51 (81 in decimal).

Again, with this tcp conn id, we can retrieve the DP associated object.

switch/Admin# show np 1 me-stats "-n 81"
Conn Policy Entry at Index: 81
-------------------------------
MSS Max: 1460  MSS Min: 0
FIN Timeout: 3600 secs  Rx Buf Share: 32768
Timewait: NONE  Nagle: Disabled
EmbryTO:   5 secs  Tx Buf Share: 32768
Rnd intial Seq: yes  Slow Strt Disabled: 1
Enque Limit: 36
SYN retry Cnt: 4  WS Factor: 0
Client Keep-Alive: 1  ACK Delay TO: 200 ms
SACK enable: 1  Timestamp enable: 1
Wind Scale Enable: 1  SYN Data Allow: 0
Server Reuse Enable: 0 Wan Opt RTT:   65535

IP Opt MIN Allow: 0  IP Opt MAX Allow: 0
IP Opt Min Clear: 0  IP Opt Max Clear: 0
IP Opt Min Deny: 0  IP Max Deny: 0
IP Opt Min Cnt: 0  IP Opt Max Cnt: 0
TCP Opt Min Clear: 1  TCP _opt Max Clear: 255
TCP Opt Min Deny: 0  TCP Opt Max Deny: 0
Norm TTL: 0  Norm TOS: 0
Norm Class: 0  Norm Hop: 0
IP Len Min: 0  IP Len Max: 0
IP Len Min Deny: 0  IP Len Max Deny: 0
Reserve Bits: 0  IP TS Action: 0 IP Rec RT Action: 0
IP Strict RT act: 0 IP Loose RT Action: 0
IP Security: 0  IP Stream: 0
IP Dont Frag: 0  Exceed MSS: 1
Chksum V: 1  TTL Ev Pr: 0
Urg: 0  Win Var: 0
TTL Norm val: 0  Class Norm Val: 0
Hop Norm Val: 0  Max Connections: 0
Inactivity TO: 4294967295 secs Unidirectional: 0
Reassemble TO: 60 secs
Conn Max: 4294967295

As can bee seen above, we have all the parameters available for our rule.

This 3rd day conclude the first part of this blog.

We covered the important commands to check that our basic configuration is correctly programmed at DP level and that we see hits on the appropriate policy.

Next week, I'll start looking into stickyness.

Average Rating: 0 (0 ratings)

Actions

Login or Register to take actions

This Blog

Posted November 22, 2010 at 1:42 AM
Stats:
Comments:0 Avg. Rating:0
Views:2701   
Shares:0

Related Content

Blogs Leaderboard