Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

6500 - route-map not used, ACL not matching traffic

Hello,

I'm performing tests with following desired scenario:

We have several remote offices, connected to our HQ via MPLS. In these remote offices, we have several vlan's. Each vlan has it's own ip-range.

The MPLS cloud is routed, so we cannot switch our HQ vlan's to the remote offices.

In this case, the client pc is in a guest vlan which allows him internet access. The uplink for this internet access is hosted in our HQ datacenter.

basic scheme:

client pc --> MPLS cloud (managed by ISP) --> 6500 switch LAN --> Checkpoint Firewall --> 6500 switch DMZ --> ASA Firewall

My test scheme:

Client pc is in a subnet A (guest vlan range office).

We receive this traffic on our first LAN 6500.

Traffic is routed from 6500 to CheckPoint (I see the client in the CheckPoint tracker connecting to external dns/http/...)

Traffic from the client to the internet is allowed in the CheckPoint.

Traffic is routed from the CheckPoint to our 6500 switch in our DMZ. While using tcpdump, I see the traffic leaving our CheckPoint on the interface that connects the CheckPoint with the 6500 in our DMZ. I even see in the dump the DST MAC is the mac of the 6500 switch.

On the 6500 in our DMZ, I've created a route-map to set the next-hop to the ASA Firewall (which is not the default route for the DMZ switch, however it is directly connected to the 6500). Only the traffic that matches an access-list gets the ip next-hop applied. The access-list contains the ip applied on the client pc, in my tcpdump on the checkpoint I see this client pc's source connecting to the internet.

The route-map is applied on the vlan-interface that connects the CheckPoint to the DMZ 6500 (which is confirmed by the matches in sequence 100).

On the 6500 in our DMZ, the route-map shows no traffic matches:

<code>

route-map vlanXXX, permit, sequence 10

  Match clauses:

    ip address (access-lists): internet-access

  Set clauses:

    ip next-hop asa.ip.addr

  Policy routing matches: 0 packets, 0 bytes

route-map vlanXXX, permit, sequence 100

  Match clauses:

  Set clauses:

  Policy routing matches: 124578711 packets, 3950495757 bytes

</code>

Anyone a clue why this isn't working?

I'm guessing: because the ACL doesn't show hits, the route-map isn't setting the next-hop. But, why doesn't this ACL match the traffic it should match?

jeroen

Everyone's tags (2)
2 ACCEPTED SOLUTIONS

Accepted Solutions

6500 - route-map not used, ACL not matching traffic

jon.marshall wrote:

Jeroen

Because you are not seeing hits does not necessarily mean the PBR is not working. On L3 switches such as the 6500 most traffic is hardware switched and when the traffic is hardware switched it does not increment the hit counter on an acl even though the acl is applied to the traffic.

So the question is whether the PBR is actually working rather than is the acl hit count being incremented. So is the traffic from the client PC being sent to the ASA ?

Jon

This might be an explanation why you are not seeing hits on an acl. I have that same experience with high-end devices like 65xx/76xx. So this means nothing.

Another possibility is this: Where do you perform NAT in order to go to the Internet?

If this is on the Checkpoint, the source address will be different and .....

regards,

Leo

Cisco Employee

6500 - route-map not used, ACL not matching traffic

By re-reading your last post and checking the initial outputs I guess I know the root cause of such punt operation.

route-map vlanXXX, permit, sequence 100

  Match clauses:

  Set clauses:

  Policy routing matches: 124578711 packets, 3950495757 bytes

this is an empty sequence number, that is a sequence without any action.

This cause the punt.

As you pointed out you need to remove it as empty sequence numbers are NOT supported and cause punt to the CPU.

Please rate and close the question if helpful or paste new outputs of show tcam after you remove sequence 100

Riccardo

29 REPLIES
Hall of Fame Super Blue

6500 - route-map not used, ACL not matching traffic

Jeroen

Because you are not seeing hits does not necessarily mean the PBR is not working. On L3 switches such as the 6500 most traffic is hardware switched and when the traffic is hardware switched it does not increment the hit counter on an acl even though the acl is applied to the traffic.

So the question is whether the PBR is actually working rather than is the acl hit count being incremented. So is the traffic from the client PC being sent to the ASA ?

Jon

Cisco Employee

6500 - route-map not used, ACL not matching traffic

Hi Jeroen,

it is not clear how the entire PBR looks like, however Jon is right. You don't get acl hits for traffic hw switched necessarily.

To check whether a PBR is working you need to check tcam stats.

i.e.

show tcam int vlan xxx acl in ip

(and 'show tcam int vlan xxx acl in ip detail' for chattier outputs)

Can you check this out and let us know?

Riccardo

New Member

6500 - route-map not used, ACL not matching traffic

Hi,

thanks for your suggestions.

Jon,

Part of my troubleshooting was: apply the same route-map on the LAN 6500.

It does show matches on the LAN 6500, but not on the DMZ 6500...

I'm quite sure the route-map causes to route in the CPU, as I see an increasing CPU load since I applied the route-map... (maybe I need to remove the sequence 100 as it matches all the other traffic)

Riccardo,

the PBR applied to that interface is only containing the rules I've pasted...

"show tcam" is an unknown command on my 6500... => enable mode solves most issues

As I expected, I have to make the same conclusion: it looks like the ACL isn't matched...

* Global Defaults shared

Entries from Bank 0

Entries from Bank 1

    permit       ip any 224.0.0.0 15.255.255.255 (28057 matches)

    policy-route ip host 10.131.246.10 any

    punt         ip any any

even the chattier output shows me: hit_cnt=0

I was expecting the ACL doesn't get matched. I expect there is no use to investigate further route-map issues? As long as I don't see any matches in my ACL, the policy will not be applied.

Still, why doesn't the ACL match the traffic it should match? I applied the identical route-map & ACL on my LAN 6500 where it does match the traffic it should match... (but it is not the right part of the network to apply the policy).

jeroen

Cisco Employee

6500 - route-map not used, ACL not matching traffic

Hi jeroen,

on the contrary you need to troubleshoot PBR issues for sure

tcam shows a final

 punt         ip any any

that indicates that traffic noy matching previous ACEs (which is forwarded according to defaut routing) is sent to the CPU.

So you were right that traffic is going to the CPU. By the way if traffic is going to the CPU you will see hits in ACL.

I guess we can solve this right away as most likely it is due to unsupported PBR entries.

Can you paste the entire route map for me?

Riccardo

Cisco Employee

6500 - route-map not used, ACL not matching traffic

By re-reading your last post and checking the initial outputs I guess I know the root cause of such punt operation.

route-map vlanXXX, permit, sequence 100

  Match clauses:

  Set clauses:

  Policy routing matches: 124578711 packets, 3950495757 bytes

this is an empty sequence number, that is a sequence without any action.

This cause the punt.

As you pointed out you need to remove it as empty sequence numbers are NOT supported and cause punt to the CPU.

Please rate and close the question if helpful or paste new outputs of show tcam after you remove sequence 100

Riccardo

New Member

6500 - route-map not used, ACL not matching traffic

Ricardo,

I must say: I only added the sequence 100 just to make sure the other traffic wouldn't be altered.

I'm guessing there is no need for it, but I didn't want to take the risk of policy routing other traffic as this 6500 passes other critical data over the same interface.

So, removing this sequence 100 (which has indeed no config), will not cause a "deny-any" on traffic not matching sequence 10?

jeroen

Cisco Employee

Re: 6500 - route-map not used, ACL not matching traffic

100% sure!

The implicit deny all that you have at the end of the PBR means that all other traffic not matching previous sequences is denied the access to the PBR logic, meaning it has to go through default routing.

This is not a security ACL where you decide what can go and what cannot. Here you interfere with the routing decision.

Please check and see.. if you have issues I will give you my number so you can call me and insult me personally!!

I saw this mistake hundreds of times in my TAC experience, dont worry!

Riccardo

New Member

6500 - route-map not used, ACL not matching traffic

Right, I do believe you (I was going to remove it anyway, it's just there for testing).

I'll change it, but not right now, still too much traffic passing the interface

Once I've changed it I'll let you know (and tag your answer as the correct one if it works )

jeroen

Cisco Employee

Re: 6500 - route-map not used, ACL not matching traffic

uhmmm, your decision is debatable but I am ok with it.

Why is it debatable? Well, by configuring an unsupported entry you are forcing traffic in sw. As you see CPU goes high but it is not just all. Hw bandwidht is infinitevely higher than sw bandwidht (do you know that you have only 1Gbps of inband rate from data plane to the RP compared to hundreds of Gbps for hw switched traffic????).

You can quickly check how close to that 1Gbps you are by issueing a 'show ibc' and checking the rx level (ibc is the inband channel to the CPU/RP). Needless to say that if you approach 1Gbps you will start seeing drops... without considering the high CPU utilization on its own which might also create control plane instability.. (don't think will be your case though... don't want to scare you too much).

So the reason for which you think it is better to postpone that at a later time 'still too much traffic passing the interface' as a matter of fact should be the reason to urge you to do it asap!

But as I wrote since we are both living in a (almost) liberal country you can do as you wish!!

Btw you might also head 50km southwards and tell me I am wrong if you think it is worth it.

Riccardo

New Member

Re: 6500 - route-map not used, ACL not matching traffic

Thanks for your concerns Riccardo, I really appreciate it!

Our uplink to the internet is only 100Mbps. I'm not worried the CPU load will kill the 6500 right now

Because we have some SLA-crazy customer, I'm just not too happy with changing the interface.

I won't drive 50km south to tell you're wrong. I'm sure you are right, I'm just waiting for a change window to confirm it...

have a nice weekend,

Jeroen

6500 - route-map not used, ACL not matching traffic

jon.marshall wrote:

Jeroen

Because you are not seeing hits does not necessarily mean the PBR is not working. On L3 switches such as the 6500 most traffic is hardware switched and when the traffic is hardware switched it does not increment the hit counter on an acl even though the acl is applied to the traffic.

So the question is whether the PBR is actually working rather than is the acl hit count being incremented. So is the traffic from the client PC being sent to the ASA ?

Jon

This might be an explanation why you are not seeing hits on an acl. I have that same experience with high-end devices like 65xx/76xx. So this means nothing.

Another possibility is this: Where do you perform NAT in order to go to the Internet?

If this is on the Checkpoint, the source address will be different and .....

regards,

Leo

New Member

6500 - route-map not used, ACL not matching traffic

Hi,

I know about NAT on the checkpoint I'm not NAT'ing. Tcpdump showed what I was expecting...

thanks anyway, I accidentaly clicked on "correct answer" for your post. Your answer is not incorrect, but I can assure that it will not solve my issue

is there an undo button for "correct answer"?

jeroen

Re: 6500 - route-map not used, ACL not matching traffic

Don't bother! It is quite ok like this.

Besides, to my knowledge rating a post and clicking "Correct Answer" are two separate actions.

I find it hard to imagine that you clicked both in error. (?)

Nevertheless I'm sorry that it wasn't the solution. It was my best guess.

Please do not forget to rate other helpful posts as well.

Thnx4rating

Leo

Cisco Employee

6500 - route-map not used, ACL not matching traffic

Please do not forget to rate other helpful posts as well.

which also means that I would not mind getting my answer rated as well and also a 'correct answe flag on my post (yes, you can flag multiple answers as the correct ones)

Cisco Employee

Re: 6500 - route-map not used, ACL not matching traffic

Hi Jeroen,

did you remove that empty sequence entry then?

Riccardo

New Member

6500 - route-map not used, ACL not matching traffic

Riccardo,

believe me or not, I'm going to alter it tonight...

I'll keep you posted.

jeroen

Cisco Employee

6500 - route-map not used, ACL not matching traffic

I do believe you, why not?! 

New Member

6500 - route-map not used, ACL not matching traffic

Nevermind

I did what you suggested, I removed the empty route-map sequence...

Traffic still flows (nice), but the matching doesn't work

the tcam cmd output:

    permit       ip any 224.0.0.0 15.255.255.255 (179 matches)

    policy-route ip host 10.131.246.10 any

    permit       ip any any

Via tcpdump I see traffic leaving my CheckPoint with src 10.131.246.10.

19:24:12.083997 10.131.246.10 > 8.8.8.8: icmp: echo request

The CheckPoint is directly connected to the DMZ 6500, I've applied the route-map on the 6500 vlan interface...

any idea? I find it strange the route-map sequence doesn't get a hit... I've applied the same route-map on another 6500 somewhere else in the network and there it did show hits...

jeroen

Cisco Employee

Re: 6500 - route-map not used, ACL not matching traffic

first notation is that the PBR is now in hw... this is what we wanted

    permit       ip any 224.0.0.0 15.255.255.255 (179 matches)

    policy-route ip host 10.131.246.10 any

    permit       ip any any

second notation that it seems that unlike our expectations this traffic is not hitting this interface, or more likely we don't have increasing counters.

You said that the PBR next hop is the ASA address. If the PBR would not work you would not reach the Internet, correct?

Or to put it simpler the PBR is apparently working.. this is what you mean when you write "Traffic still flows (nice)", right?

I just need to be sure whether is just a counter hits issue or the PBR is not working at all.

Which IOS is this box running?

and the other one where everything works fine?

Riccardo

New Member

6500 - route-map not used, ACL not matching traffic

Ricardo,

it's true, I'm now routing in hw instead of cpu. This is indeed what I wanted.

if the PBR doesn't work, there is access to the internet but not via the right link We have multiple connections to the internet, I want to use the internet connection behind the ASA for this connection (because I don't want to consume more bandwidth on our BGP uplinks, as this might impact customer connecetions).

What I mean by "Traffic still flows" is that ALL the other traffic that should not be PB-routed is still using the right uplink (not the ASA)

funny thing, there is one hit on the route-map... I remember that one hit showed up when I altered the ACL that is used to match.

  Policy routing matches: 1 packets, 77 bytes

IOS on the 6500 where it's NOT working: s72033-adventerprisek9_wan-vz.122-33.SXI6.bin

IOS on the 6500 where it does "work" (that is: hits on route-map): s72033-ipservicesk9_wan-vz.122-33.SXH4.bin

they both are WS-C6509-E and have 2x VS-S720-10G-3C

jeroen

Cisco Employee

6500 - route-map not used, ACL not matching traffic

Jeroen,

ok I see.

trivial question, are you sure that the traffic from the firewall gets to the DMZ6500 in the vlan where you applied the PBR?

I am asking as we don't see any hits in the 'permit ip any any' ACE either... this seems to indicate that this SVI is not routing the traffic as you are expecting as it just does not receive it.

Can you verify the routing (if enabled) on the firewall to be sure that it routes this traffic to this vlan?

We might have another way to check this out which is an ELAM capture on the DMZ6500 to be 100% that the traffic enters the devica via the right interface and consequently is routed according to the PBR. But for that I need more info about your configuration and your hardware.

Riccardo

New Member

6500 - route-map not used, ACL not matching traffic

Riccardo,

the answer to your trivial question: yes I'm sure the traffic from the firewall gets to the DMZ6500 in the vlan where I applied the PBR.

Reason for being sure: the checkpoint has no routing protocol enabled. It has it's default routed pointed to the DMZ6500. The next-hop, is the other end of the VLAN (on the DMZ6500) where I applied the route-map.

Another reason for being sure: tcpdump -e (and -i to match the right outgoing interface)

In this dump, I see:

  • source IP (10.131.246.10) connecting to
  • destination IP 8.8.8.8 (I'm using google DNS for testing on this client machine)
  • SRC MAC is the checkpoint,
  • DST MAC is a mac belonging to the DMZ6500.

This DST mac shows the MAC belongs to the router for the correct vlan on the DMZ6500.

Never heard of an ELAM...?

regards,

Jeroen

New Member

6500 - route-map not used, ACL not matching traffic

Riccardo,

I think I focussed too much on the route-map not showing hits...

It works now, there was an issue with a route on the ASA firewall to return the replied traffic...

However, the route-map is still not showing any hits. Is this normal behaviour?

Jeroen

Cisco Employee

6500 - route-map not used, ACL not matching traffic

Hi Jeroen,

well yes, the route-map does not show any hits unless it forwards traffic in sw (the RP engine); but this is something that we don't want and like.

Source of infromation is the tcam as we discussed days ago. If you see hits in show tcam you are fine.

Btw ELAM is an engineering command meant to capture a defined packet of a given flow (through the use of ad-hoc filters) and check the actual forwarding decision taken by the PFC or DFC which handled it. and much more....

I was ready to disclose the secret fo you but apparently thre is no need 

Riccardo

New Member

6500 - route-map not used, ACL not matching traffic

Hold your horses... there might still be a need to disclose the secret

Entries from Bank 1

    permit       ip any 224.0.0.0 15.255.255.255 (134760 matches)

    policy-route ip host 10.131.246.10 any

    permit       ip any any

route-map vlanxxx, permit, sequence 10

  Match clauses:

    ip address (access-lists): plain-internet-access

  Set clauses:

    ip next-hop asa.ip.addr

  Policy routing matches: 1 packets, 77 bytes

There is no difference between now (see above) and last week.

I don't see any hit on the tcam output, nor on the route-map output (except from the 1 hit that showed up when I altered the ACL). This isn't a displaybug?

However, there is no other way for the ASA to see any hits from my source IP unless the set next-hop was excecuted.

It seems to work, but it seems impossible to confirm this by the known display commands on the DMZ6500...

Jeroen

Cisco Employee

6500 - route-map not used, ACL not matching traffic

indeed it seems some kind of display bug as you confirm that traffic is actually policy routed.

I would first try to remove and re-apply the pbr on that svi to see if counters start moving afterwards.

After that if the issue is still there we might play with some 'magic' commands  and see what's going on internally 

Riccardo

New Member

6500 - route-map not used, ACL not matching traffic

Riccardo,

I did already try the removal & re-apply of the pbr while troubleshooting. It showed no result.

The 1 hit that matched visually the pbr showed up while altering the ACL.

This isn't a known bug in my IOS?

I think your magic commands will most likely cause service interruption?

jeroen

Cisco Employee

6500 - route-map not used, ACL not matching traffic

Jeroen,

on top of my head I am not aware of any bug affecting SXI6 giving similar symptoms... but I could be wrong. Moreover you are running Modular IOS (ION) which is in end of life and sometimes is affected by problems not affecting plain IOS.

I think your magic commands will most likely cause service interruption?

not even a microsecond!!! Elam it is a an extremely safe way to investigate this platform

if you want to go for it I need few extra info upfront:

- show module

- show run interface for both the phisical port and the SVI which receives traffic from the firewall

- same info for the physical port and the SVI towards the ASA (destination).

If you don't want to disclose this info in public feel free to send me pvt message

I will reply with the procedure and the exact filter we need for the elam capture

Riccardo

New Member

6500 - route-map not used, ACL not matching traffic

Hi Riccardo,

although I'm interested in the ELAM tracing, I don't see it useful for this issue. It works right now, but the counters don't increase I can live with that for the moment. Perhaps if I'll upgrade the IOS in the future, this will be solved.

Thanks for you explanations (especially by removing the empty sequence).

jeroen

5187
Views
0
Helpful
29
Replies
CreatePlease login to create content