Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Webcast-Catalyst9k
New Member

CEF broken for packets going through 'ip nat inside' VLAN interface

Hi -

One of the more odd issues I've encountered with Cisco and I have a TAC case open now. Here's the issue:

IP Transit via static route

            ||

Cisco 6509E Sup720 BXL w/ 3x WS-X6748-GE-TX <> Cisco 6509E Sup720 BXL w/ 3x WS-X6748-GE-TX

            ||

Internal hosts on 10.200.4.0/22 network

When i'm on the 6509, I can source ping from the 10.200.4.2 VLAN (VLAN 44) IP to external Internet just fine. We can check nat translations and see this, so we know nat is set up ok from that subnet:

core1-22002#ping 4.2.2.2 source vlan 44

!!!!!

core1-22001#sh ip nat translations | i 4.2.2.2

icmp 141.136.100.122:1024 10.200.4.2:124        4.2.2.2:124           4.2.2.2:1024

Here is the ACL being referenced for NAT rule:

Extended IP access list 118

    10 permit ip 10.200.0.0 0.0.255.255 any log (9 matches) << matches coming from source ping tests

However, when a server connected to VLAN 44 (10.200.4.x/22) tries to get out, it fails without NAT ever trying to set up the translation and doesn't add log match to entry. I should say that these hosts can ping across subnets internally, reach the public VLAN gateway, and reach any other IP on the 6509 or directly connected subnets.

So I kept saying to myself, its almost as if the VLAN is just dropping the packet. So on the phone with Cisco TAC, after about 1 hour of verification on config (everything was correct), he modified the configuration from this:

interface Vlan44

ip address 10.200.4.2 255.255.252.0

ip nat inside

to this:

core1-22002(config)#ip access-l ex test

core1-22002(config-ext-nacl)#10 permit ip host 10.200.4.36 any log

core1-22002(config-ext-nacl)#20 permit ip any any

core1-22002(config)#int vlan 44

core1-22002(config-if)#ip access-g test in

and BOOM, it works

Extended IP access list test

    10 permit ip host 10.200.4.36 any log (12 matches)

    20 permit ip any any

Eventually we figure out ANY ACL permit statement that matches works, as long as the "log" command is entered on ACL AND ACL is applied directly to that VLAN interface!

The Cisco TAC tells me that the 'log' command forces process switching and basically means that CEF is not working in this case, so the only way packets can forward out of the interface to the outside world is to use process switching instead of fast switching/CEF. Obviously something is wrong here and no way am I putting our large amount of PPS prod network on something that has to touch the CPU/higher level resources every time we need to forward a packet.

Has anyone ever seen this? It seems like an IOS change may be in order, fix it, but wanted to run it by here first. Both 6509's (connected via 2x 10G port-channel) display the exact same behavior. TAC punted me to switching group from routing group and of course I'm still waiting to hear back. Thanks in advance - Jason

--------------------------------------

CONFIG:

core1-22001#sh ver

Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXI9, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2012 by Cisco Systems, Inc.

Compiled Fri 24-Feb-12 21:38 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)

core1-22001#sh module

Mod Ports Card Type                              Model              Serial No.

--- ----- -------------------------------------- ------------------ -----------

  1   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAD10450C3P

  2   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL08311AL6

  3   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL08311ALU

  5    2  Supervisor Engine 720 (Active)         WS-SUP720-3BXL     SAD110400EG

  6    0  Unknown (Other)                        Unknown            Unknown

  7    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      SAD104803EN

  8   16  SFM-capable 16 port 1000mb GBIC        WS-X6516-GBIC      SAL0730HEKR

  9   16  SFM-capable 16 port 1000mb GBIC        WS-X6516-GBIC      SAL0739M7TA

Mod MAC addresses                       Hw    Fw           Sw           Status

--- ---------------------------------- ------ ------------ ------------ -------

  1  0019.aacc.6692 to 0019.aacc.66c1   2.4   12.2(14r)S5  12.2(33)SXI9 Ok

  2  0013.60ff.4e50 to 0013.60ff.4e7f   2.0   12.2(14r)S5  12.2(33)SXI9 Ok

  3  0013.60ff.2920 to 0013.60ff.294f   2.1   12.2(14r)S5  12.2(33)SXI9 Ok

  5  0013.807b.49e8 to 0013.807b.49eb   5.3   8.4(2)       12.2(33)SXI9 Ok

  6  0000.0000.0000 to 0000.0000.0000   0.0   Unknown      Unknown      Unknown

  7  001a.2f00.1380 to 001a.2f00.1383   2.4   12.2(14r)S5  12.2(33)SXI9 Ok

  8  0008.7dc9.ce50 to 0008.7dc9.ce5f   5.5   Unknown      12.2(33)SXI9 PwrDown

  9  000d.bc3b.da60 to 000d.bc3b.da6f   5.5   6.3(1)       12.2(33)SXI9 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status

---- --------------------------- ------------------ ----------- ------- -------

  1  Centralized Forwarding Card WS-F6700-CFC       SAL14502G4C  4.1    Ok

  2  Centralized Forwarding Card WS-F6700-CFC       SAD074104SA  1.1    Ok

  3  Centralized Forwarding Card WS-F6700-CFC       SAL09444N7C  2.0    Ok

  5  Policy Feature Card 3       WS-F6K-PFC3BXL     SAD110302YT  1.8    Ok

  5  MSFC3 Daughterboard         WS-SUP720          SAD110200P3  2.6    Ok

  7  Centralized Forwarding Card WS-F6700-CFC       SAL1249C4MD  4.1    Ok

Mod  Online Diag Status

---- -------------------

  1  Pass

  2  Pass

  3  Pass

  5  Pass

  6  Not Applicable

  7  Pass

  8  Not Applicable

  9  Pass

core1-22001#

core1-22001#sh run

Building configuration...

Current configuration : 18013 bytes

!

! Last configuration change at 21:44:54 UTC Wed Jul 17 2013 by pubmatic_ops

!

upgrade fpd auto

version 12.2

service timestamps debug uptime

service timestamps log uptime

no service password-encryption

service counters max age 5

!

hostname core1-22001

!

boot-start-marker

boot system flash disk0:s72033-advipservicesk9_wan-mz.122-33.SXI9.bin

boot-end-marker

!

logging console warnings

enable secret 5

!

username

aaa new-model

!

!

aaa authentication login default local

!

!

!

aaa session-id common

logging event link-status default

!

!

!

ip ssh time-out 60

ip ssh authentication-retries 5

no ip domain-lookup

ip domain-name pubmatic.com

ip name-server 208.67.222.222

ip name-server 8.8.8.8

ipv6 mfib hardware-switching replication-mode ingress

udld enable

no mls acl tcam share-global

mls netflow interface

mls cef error action freeze

!

!

!

!

!

!

!

!

!

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

diagnostic bootup level minimal

access-list 24 permit 10.200.0.0 0.0.255.255 log

access-list 118 permit ip 10.200.0.0 0.0.255.255 any log

!

redundancy

main-cpu

  auto-sync running-config

mode sso

!

ip access-list extended inside_nat_acl

permit ip 10.200.1.0 0.0.0.255 198.47.127.0 0.0.0.255 log

ip access-list extended test

permit ip any any log

!

vlan internal allocation policy ascending

vlan access-log ratelimit 2000

!

!

!

!

!

interface Loopback0

no ip address

!

interface Loopback198

no ip address

!

interface Port-channel2

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface Port-channel16

switchport

switchport access vlan 11

switchport mode access

interface GigabitEthernet1/48

description test port

switchport

switchport access vlan 22

switchport mode access

interface GigabitEthernet2/47

switchport

switchport access vlan 44

switchport mode access

spanning-tree portfast edge

!

interface GigabitEthernet3/1

no ip address

!

interface GigabitEthernet3/2

no ip address

!

interface GigabitEthernet3/3

no ip address

!

interface GigabitEthernet3/4

no ip address

!

interface GigabitEthernet3/5

no ip address

!

interface GigabitEthernet3/6

no ip address

!

interface GigabitEthernet3/7

no ip address

!

interface GigabitEthernet3/8

no ip address

!

interface GigabitEthernet3/9

no ip address

!

interface GigabitEthernet3/10

description xcp22006-pub

switchport

switchport access vlan 100

switchport mode access

!

interface TenGigabitEthernet7/1

ip address 141.136.102.122 255.255.255.252

ip nat outside

!

interface TenGigabitEthernet7/2

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 2 mode active

!

interface TenGigabitEthernet7/3

switchport

switchport access vlan 100

switchport mode access

!

interface TenGigabitEthernet7/4

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 2 mode active

!

interface Vlan1

no ip address

!

interface Vlan2

ip address 10.200.0.1 255.255.255.0 secondary

ip address 10.200.0.2 255.255.255.0

no ip redirects

ip nat inside

standby 1 name pub-ams-net1

!

interface Vlan11

ip address 10.200.1.2 255.255.255.0

ip access-group test in

ip nat inside

standby 11 ip 10.200.1.1

!

interface Vlan22

ip address 10.200.2.1 255.255.255.0 secondary

ip address 10.200.2.2 255.255.255.0

no ip redirects

ip nat inside

!

interface Vlan44

ip address 10.200.4.2 255.255.252.0

ip access-group test in

ip nat inside

standby 44 ip 10.200.4.1

!

interface Vlan100

ip address 198.47.127.1 255.255.255.0

ip nat outside

!

interface Vlan101

ip address 10.200.254.2 255.255.255.0

!

interface Vlan999

ip address 10.200.99.1 255.255.255.0

!

ip nat pool private_outbound 198.47.127.118 198.47.127.120 prefix-length 24

ip nat inside source list 118 interface TenGigabitEthernet7/1 overload

ip classless

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 141.136.102.121

!

!

no ip http server

no ip http secure-server

!

!

!

!

control-plane

!

!

dial-peer cor custom

!

!

!

!

line con 0

exec-timeout 0 0

line vty 0 4

transport input ssh

!

!

end

core1-22001#

2 REPLIES
Cisco Employee

Re: CEF broken for packets going through 'ip nat inside' VLAN in

Good morning Jason,

Could you provide the service request number please?

When you say that these hosts can ping across subnets internally, can I correctly assume that interVLAN routing works fine? This would still be flow through, hardware switched traffic from the switch's perspective, so it looks like there are very specific flows that are not being correctly hardware switched.

To begin with, for a traffic flow that is failing, could I get the outputs of the following please:

1. show ip route and the show ip route output for any recursive routes.

2. show ip cef exact-route

3. show mls cef exact-route

Regards,

Aninda

Message was edited by: Aninda Chatterjee

New Member

CEF broken for packets going through 'ip nat inside' VLAN interf

For search and troubleshooting for others who may experience this, below is the resolution with TAC's help.

We observed the following:

+ routed traffic for host was being dropped on the 6509 with NAT configured

+ with an explicit permit ACL directly applied to internal VLAN interface we could get the traffic to flow through the box since it was punted to the cpu for processing

+ ELAM and NETDR capture showed that without the ACL traffic was still punted to the CPU, however all packets were still getting dropped

+ we also saw that applying the ACL, starting a ping (which worked), and removing the ACL from the interface did not kill off the traffic (icmp continued to flow from host to destination)

+ ALL cef entries looked good as suggested by Aninda

We identified the following;

access-list 118 permit ip 10.200.0.0 0.0.255.255 any log   <<<

ip nat pool private_outbound 19.247.127.118 19.247.127.120 prefix-length 24

ip nat inside source list 100 interface Vlan100 overload

ip nat inside source list 118 pool private_outbound overload

due to the log keyword on the ACL 118, this 'log' was causing the traffic to be punted to the CPU and and getting drop at the cpu, and the result was that we were not getting NAT translation entries.

There was also a missing netflow flowmask configured. NAT requires an interface full configuration.

After removing the log keyword service were restored.

1346
Views
0
Helpful
2
Replies
CreatePlease to create content