FWSM 3.1(4) - Missing ACL element

Mel Popple · ‎07-23-2010

Has anyone encountered missing elements in their FWSM ACLs before? We have an issue where an ACL rule, that uses an object-group, has decided to miss out one of the elements in the ACL even though it has got it correct in rules before and after the offending one!

The FWSM is in multiple context mode and the resource partitions are set at the defaults and are nowhere near reaching their ACL element limits. The context and FWSM is happy in every way except for missing this one element at the end of line 66 (indicated in lines 57 & 85 by the arrow). The group is used in about 10 rules altogether in this context but only the element in line 66 is missing.

FW/CONTEXT1# sh access-list CSM_FW_ACL_INT-1 | i line 57
access-list CSM_FW_ACL_INT-1 line 57 extended permit ip object-group Internal-Networks object-group SVR_GRP1 0x92823732
access-list CSM_FW_ACL_INT-1 line 57 extended permit ip xxx.xxx.0.0 255.255.128.0 xxx.xxx.246.0 255.255.255.0 (hitcnt=338988) 0xe136ae0d
access-list CSM_FW_ACL_INT-1 line 57 extended permit ip xxx.xxx.128.0 255.255.192.0 xxx.xxx.246.0 255.255.255.0 (hitcnt=586) 0xd630d7a2
access-list CSM_FW_ACL_INT-1 line 57 extended permit ip xxx.xxx.192.0 255.255.224.0 xxx.xxx.246.0 255.255.255.0 (hitcnt=910) 0xaa2c2aa8
access-list CSM_FW_ACL_INT-1 line 57 extended permit ip xxx.xxx.224.0 255.255.240.0 xxx.xxx.246.0 255.255.255.0 (hitcnt=1055) 0x19f4d7e8
access-list CSM_FW_ACL_INT-1 line 57 extended permit ip yyy.yyy.0.0 255.252.0.0 xxx.xxx.246.0 255.255.255.0 (hitcnt=0) 0x3ba7a923
access-list CSM_FW_ACL_INT-1 line 57 extended permit ip zzz.zzz.11.0 255.255.255.0 xxx.xxx.246.0 255.255.255.0 (hitcnt=66) 0x6b4b7ef4
access-list CSM_FW_ACL_INT-1 line 57 extended permit ip aaa.aaa.0.0 255.252.0.0 xxx.xxx.246.0 255.255.255.0 (hitcnt=0) 0xc8c3ab0a <------
FW/CONTEXT1# sh access-list CSM_FW_ACL_INT-1 | i line 66
access-list CSM_FW_ACL_INT-1 line 66 extended permit tcp object-group Internal-Networks object-group SVR_GRP2 eq ftp 0x4944127d
access-list CSM_FW_ACL_INT-1 line 66 extended permit tcp xxx.xxx.0.0 255.255.128.0 host xxx.xxx.251.2 eq ftp (hitcnt=300780) 0x409631b4
access-list CSM_FW_ACL_INT-1 line 66 extended permit tcp xxx.xxx.128.0 255.255.192.0 host xxx.xxx.251.2 eq ftp (hitcnt=929) 0xd0cf997d
access-list CSM_FW_ACL_INT-1 line 66 extended permit tcp xxx.xxx.192.0 255.255.224.0 host xxx.xxx.251.2 eq ftp (hitcnt=0) 0xf2ecc647
access-list CSM_FW_ACL_INT-1 line 66 extended permit tcp xxx.xxx.224.0 255.255.240.0 host xxx.xxx.251.2 eq ftp (hitcnt=2) 0xacf1194
access-list CSM_FW_ACL_INT-1 line 66 extended permit tcp yyy.yyy.0.0 255.252.0.0 host xxx.xxx.251.2 eq ftp (hitcnt=0) 0x8a643a5a
access-list CSM_FW_ACL_INT-1 line 66 extended permit tcp zzz.zzz.11.0 255.255.255.0 host xxx.xxx.251.2 eq ftp (hitcnt=0) 0x87411ddc
FW/CONTEXT1# sh access-list CSM_FW_ACL_INT-1 | i line 85
access-list CSM_FW_ACL_INT-1 line 85 extended permit tcp object-group Internal-Networks object-group SVR_GRP3 eq https 0x4866c7ed
access-list CSM_FW_ACL_INT-1 line 85 extended permit tcp xxx.xxx.0.0 255.255.128.0 host xxx.xxx.251.5 eq https (hitcnt=0) 0xdd8e2cb2
access-list CSM_FW_ACL_INT-1 line 85 extended permit tcp xxx.xxx.128.0 255.255.192.0 host xxx.xxx.251.5 eq https (hitcnt=0) 0x4b9b6140
access-list CSM_FW_ACL_INT-1 line 85 extended permit tcp xxx.xxx.192.0 255.255.224.0 host xxx.xxx.251.5 eq https (hitcnt=0) 0x9d4a5ce6
access-list CSM_FW_ACL_INT-1 line 85 extended permit tcp xxx.xxx.224.0 255.255.240.0 host xxx.xxx.251.5 eq https (hitcnt=0) 0xcee3e297
access-list CSM_FW_ACL_INT-1 line 85 extended permit tcp yyy.yyy.0.0 255.252.0.0 host xxx.xxx.251.5 eq https (hitcnt=0) 0xd212d608
access-list CSM_FW_ACL_INT-1 line 85 extended permit tcp zzz.zzz.11.0 255.255.255.0 host xxx.xxx.251.5 eq https (hitcnt=0) 0x601fbc
access-list CSM_FW_ACL_INT-1 line 85 extended permit tcp aaa.aaa.0.0 255.252.0.0 host xxx.xxx.251.5 eq https (hitcnt=0) 0x75894ff8 <------
FW/CONTEXT1# sh access-list | i element
access-list CSM_FW_ACL_INT-1; 1967 elements
access-list CSM_FW_ACL_INT-2; 2287 elements
access-list CSM_FW_ACL_INT-3; 2 elements

object-group network Internal-Networks
network-object xxx.xxx.0.0 255.255.128.0
network-object xxx.xxx.128.0 255.255.192.0
network-object xxx.xxx.192.0 255.255.224.0
network-object xxx.xxx.224.0 255.255.240.0
network-object yyy.yyy.0.0 255.252.0.0
network-object zzz.zzz.11.0 255.255.255.0
network-object aaa.aaa.0.0 255.252.0.0

Since we are using CSM we will disable the rule and push the policy, then enable the rule and push the policy again - hopefully that will fix the problem. CSM can't be the fault on this one as the object group is used on multiple context on this FWSM and also on other ASAs all with no issues.

Anyone seen this before?

Regards

Mel

Panos Kampanakis · ‎07-23-2010

Can you change the ACL and have it recompile (you said you will do it with CSM)? Does the ACE show up?

I think it will. If it does then it is an ACL corruption defect tha existed in early FWSM 3.1 versions.

Please update the thread after your test for reference of other people in the future.

Rgs,

PK

Magnus Mortensen · ‎07-23-2010

Mel,

If you check the full ACL is there any chance that the same rule may be in the ACL at a different place (not part of the object group?). What do you get if you issue the command "show access-list CSM_FW_ACL_INT-1 | in aaa.aaa.0.0"? If the same is created as part of an object group and it overlaps with another line we may see some interesting results..

- Magnus

Magnus Mortensen · ‎07-23-2010

Mel,

I did a little more digging on my hunch from the original reply. A code change was put into 3.1.4 to check to see if there were any duplicate entries as a result of an overlap between objectgroup lines and other lines already present in the config (the goal was to save space in ACL memeory by not programming duplicate entries). As a result of that code change, I think, 3.1.4 will ignore the duplicate when it i builds the ACL (Issue a warning too! Which should have been in the CSM transcript of the deployment... In theory) . This may be the issue you are seeing. This code change caused some other problems and was removed again in later code. Version 3.1.5 does not have this code change allows for the duplicate entries and should show up just fine. But all of that hinges on a hunch that there is some overlap in your config. I am guessing there is an ACL line with a lower line number than 66 that reads:

access-list CSM_FW_ACL_INT-1 line ## extended permit tcp aaa.aaa.0.0 255.252.0.0 host xxx.xxx.251.2 eq ftp

Now if that doesn't show up, then were back at square one...

- Magnus

Mel Popple · ‎07-26-2010

Magnus,

Thanks for your time and input on this. Sadly we are back to square one.

I have checked the ACL by the method you suggest and also used CSM's rule analysis tool and it doesn't show it as being a duplicate. We have a change scheduled for this Wednesday to try and correct this issue so will post the results after that.

Regards

Mel

Magnus Mortensen · ‎07-26-2010

Mel,

At this point it may make sense to spawn a Service Request for this. If you are runningo into an ACL corruption issue, you probably want to find the root cause since ACL corruption could turn into a security problem. If you do open a ticket, let the engineer know aout this thread.

- Magnus

Mel Popple · ‎07-28-2010

Disabling the rule in CSM and pushing the config, then enabling the rule and pushing the config again cured the problem. Looks like something must not have quite sorted itself out correctly the last time we pushed the config even though it had indicated it was successful.

Thanks for all the advice and help offered.

Mel