6500 - route-map not used, ACL not matching traffic

[Jeroen Huysmans]

Hello,

I'm performing tests with following desired scenario:

We have several remote offices, connected to our HQ via MPLS. In these remote offices, we have several vlan's. Each vlan has it's own ip-range.

The MPLS cloud is routed, so we cannot switch our HQ vlan's to the remote offices.

In this case, the client pc is in a guest vlan which allows him internet access. The uplink for this internet access is hosted in our HQ datacenter.

basic scheme:

client pc --> MPLS cloud (managed by ISP) --> 6500 switch LAN --> Checkpoint Firewall --> 6500 switch DMZ --> ASA Firewall

My test scheme:

Client pc is in a subnet A (guest vlan range office).

We receive this traffic on our first LAN 6500.

Traffic is routed from 6500 to CheckPoint (I see the client in the CheckPoint tracker connecting to external dns/http/...)

Traffic from the client to the internet is allowed in the CheckPoint.

Traffic is routed from the CheckPoint to our 6500 switch in our DMZ. While using tcpdump, I see the traffic leaving our CheckPoint on the interface that connects the CheckPoint with the 6500 in our DMZ. I even see in the dump the DST MAC is the mac of the 6500 switch.

On the 6500 in our DMZ, I've created a route-map to set the next-hop to the ASA Firewall (which is not the default route for the DMZ switch, however it is directly connected to the 6500). Only the traffic that matches an access-list gets the ip next-hop applied. The access-list contains the ip applied on the client pc, in my tcpdump on the checkpoint I see this client pc's source connecting to the internet.

The route-map is applied on the vlan-interface that connects the CheckPoint to the DMZ 6500 (which is confirmed by the matches in sequence 100).

On the 6500 in our DMZ, the route-map shows no traffic matches:

<code>

route-map vlanXXX, permit, sequence 10

  Match clauses:

    ip address (access-lists): internet-access

  Set clauses:

    ip next-hop asa.ip.addr

  Policy routing matches: 0 packets, 0 bytes

route-map vlanXXX, permit, sequence 100

  Match clauses:

  Set clauses:

  Policy routing matches: 124578711 packets, 3950495757 bytes

</code>

Anyone a clue why this isn't working?

I'm guessing: because the ACL doesn't show hits, the route-map isn't setting the next-hop. But, why doesn't this ACL match the traffic it should match?

jeroen

[Jon Marshall]

Jeroen

Because you are not seeing hits does not necessarily mean the PBR is not working. On L3 switches such as the 6500 most traffic is hardware switched and when the traffic is hardware switched it does not increment the hit counter on an acl even though the acl is applied to the traffic.

So the question is whether the PBR is actually working rather than is the acl hit count being incremented. So is the traffic from the client PC being sent to the ASA ?

Jon

[rsimoni]

Hi Jeroen,

it is not clear how the entire PBR looks like, however Jon is right. You don't get acl hits for traffic hw switched necessarily.

To check whether a PBR is working you need to check tcam stats.

i.e.

show tcam int vlan xxx acl in ip

(and 'show tcam int vlan xxx acl in ip detail' for chattier outputs)

Can you check this out and let us know?

Riccardo

[Jeroen Huysmans]

Hi,

thanks for your suggestions.

Jon,

Part of my troubleshooting was: apply the same route-map on the LAN 6500.

It does show matches on the LAN 6500, but not on the DMZ 6500...

I'm quite sure the route-map causes to route in the CPU, as I see an increasing CPU load since I applied the route-map... (maybe I need to remove the sequence 100 as it matches all the other traffic)

Riccardo,

the PBR applied to that interface is only containing the rules I've pasted...

"show tcam" is an unknown command on my 6500... => enable mode solves most issues

As I expected, I have to make the same conclusion: it looks like the ACL isn't matched...

* Global Defaults shared

Entries from Bank 0

Entries from Bank 1

    permit       ip any 224.0.0.0 15.255.255.255 (28057 matches)

    policy-route ip host 10.131.246.10 any

    punt         ip any any

even the chattier output shows me: hit_cnt=0

I was expecting the ACL doesn't get matched. I expect there is no use to investigate further route-map issues? As long as I don't see any matches in my ACL, the policy will not be applied.

Still, why doesn't the ACL match the traffic it should match? I applied the identical route-map & ACL on my LAN 6500 where it does match the traffic it should match... (but it is not the right part of the network to apply the policy).

jeroen

[rsimoni]

Hi jeroen,

on the contrary you need to troubleshoot PBR issues for sure

tcam shows a final

 punt         ip any any

that indicates that traffic noy matching previous ACEs (which is forwarded according to defaut routing) is sent to the CPU.

So you were right that traffic is going to the CPU. By the way if traffic is going to the CPU you will see hits in ACL.

I guess we can solve this right away as most likely it is due to unsupported PBR entries.

Can you paste the entire route map for me?

Riccardo

[rsimoni · Accepted solution]

By re-reading your last post and checking the initial outputs I guess I know the root cause of such punt operation.

route-map vlanXXX, permit, sequence 100
  Match clauses:
  Set clauses:
  Policy routing matches: 124578711 packets, 3950495757 bytes

this is an empty sequence number, that is a sequence without any action.

This cause the punt.

As you pointed out you need to remove it as empty sequence numbers are NOT supported and cause punt to the CPU.

Please rate and close the question if helpful or paste new outputs of show tcam after you remove sequence 100

Riccardo

[Jeroen Huysmans]

Ricardo,

I must say: I only added the sequence 100 just to make sure the other traffic wouldn't be altered.

I'm guessing there is no need for it, but I didn't want to take the risk of policy routing other traffic as this 6500 passes other critical data over the same interface.

So, removing this sequence 100 (which has indeed no config), will not cause a "deny-any" on traffic not matching sequence 10?

jeroen

[rsimoni]

100% sure!

The implicit deny all that you have at the end of the PBR means that all other traffic not matching previous sequences is denied the access to the PBR logic, meaning it has to go through default routing.

This is not a security ACL where you decide what can go and what cannot. Here you interfere with the routing decision.

Please check and see.. if you have issues I will give you my number so you can call me and insult me personally!!

I saw this mistake hundreds of times in my TAC experience, dont worry!

Riccardo

[Jeroen Huysmans]

Right, I do believe you (I was going to remove it anyway, it's just there for testing).

I'll change it, but not right now, still too much traffic passing the interface

Once I've changed it I'll let you know (and tag your answer as the correct one if it works )

jeroen

[rsimoni]

uhmmm, your decision is debatable but I am ok with it.

Why is it debatable? Well, by configuring an unsupported entry you are forcing traffic in sw. As you see CPU goes high but it is not just all. Hw bandwidht is infinitevely higher than sw bandwidht (do you know that you have only 1Gbps of inband rate from data plane to the RP compared to hundreds of Gbps for hw switched traffic????).

You can quickly check how close to that 1Gbps you are by issueing a 'show ibc' and checking the rx level (ibc is the inband channel to the CPU/RP). Needless to say that if you approach 1Gbps you will start seeing drops... without considering the high CPU utilization on its own which might also create control plane instability.. (don't think will be your case though... don't want to scare you too much).

So the reason for which you think it is better to postpone that at a later time 'still too much traffic passing the interface' as a matter of fact should be the reason to urge you to do it asap!

But as I wrote since we are both living in a (almost) liberal country you can do as you wish!!

Btw you might also head 50km southwards and tell me I am wrong if you think it is worth it.

Riccardo

[Jeroen Huysmans]

Thanks for your concerns Riccardo, I really appreciate it!

Our uplink to the internet is only 100Mbps. I'm not worried the CPU load will kill the 6500 right now

Because we have some SLA-crazy customer, I'm just not too happy with changing the interface.

I won't drive 50km south to tell you're wrong. I'm sure you are right, I'm just waiting for a change window to confirm it...

have a nice weekend,

Jeroen

[lgijssel · Accepted solution]

jon.marshall wrote:
Jeroen
Because you are not seeing hits does not necessarily mean the PBR is not working. On L3 switches such as the 6500 most traffic is hardware switched and when the traffic is hardware switched it does not increment the hit counter on an acl even though the acl is applied to the traffic. 
So the question is whether the PBR is actually working rather than is the acl hit count being incremented. So is the traffic from the client PC being sent to the ASA ? 
Jon

This might be an explanation why you are not seeing hits on an acl. I have that same experience with high-end devices like 65xx/76xx. So this means nothing.

Another possibility is this: Where do you perform NAT in order to go to the Internet?

If this is on the Checkpoint, the source address will be different and .....

regards,

Leo

[Jeroen Huysmans]

Hi,

I know about NAT on the checkpoint I'm not NAT'ing. Tcpdump showed what I was expecting...

thanks anyway, I accidentaly clicked on "correct answer" for your post. Your answer is not incorrect, but I can assure that it will not solve my issue

is there an undo button for "correct answer"?

jeroen

[lgijssel]

Don't bother! It is quite ok like this.

Besides, to my knowledge rating a post and clicking "Correct Answer" are two separate actions.

I find it hard to imagine that you clicked both in error. (?)

Nevertheless I'm sorry that it wasn't the solution. It was my best guess.

Please do not forget to rate other helpful posts as well.

Thnx4rating

Leo

[rsimoni]

Please do not forget to rate other helpful posts as well.

which also means that I would not mind getting my answer rated as well and also a 'correct answe flag on my post (yes, you can flag multiple answers as the correct ones)