Solved: FWSM performance issue

madhusudhan s · ‎01-11-2010

Hi all,

Our customer's Datacentre live on Saturday. Since then we have faced major performance issue in FWSM. The latency jumps up whenever we access the CLI or add any entries to the device. The memory utilization is already 49% even though we have placed only 1000 ACL entries as of now.customer suspect major performance degradation once Full Load is there on the Device. customer have around 10,000 ACL entries to be added. kindly suggets

regards

Madhu

attached the logs and sh version output.

============================

sh ver

FWSM Firewall Version 3.1(10)

Device Manager Version 6.1(5)F

Compiled on Mon 21-Apr-08 17:43 by fwsmbld

Religare-FWSM up 2 days 11 hours

failover cluster up 2 days 11 hours

Hardware: WS-SVC-FWM-1, 1024 MB RAM, CPU Pentium III 1000 MHz

Flash STI Flash 8.0.0 @ 0xc321, 20MB

0: Int: Not licensed : irq 5

1: Int: Not licensed : irq 7

2: Int: Not licensed : irq 11

The Running Activation Key is not set, using default settings:

Licensed features for this platform:

Maximum Interfaces : 256

Inside Hosts : Unlimited

Failover : Active/Active

VPN-DES : Enabled

VPN-3DES-AES : Enabled

Cut-through Proxy : Enabled

Guards : Enabled

URL Filtering : Enabled

Security Contexts : 2

GTP/GPRS : Disabled

VPN Peers : Unlimited

Serial Number: SAD125004FT

Running Activation Key: 0x00000000 0x00000000 0x00000000 0x00000000

Configuration last modified by enable_15 at 14:41:28.611 IST Mon Jan 11

2010

============================================

Sh logging

sh lof gg

FWSM-Switch-Primary>sh loggi

Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

Console logging: level debugging, 92 messages logged, xml disabled,

filtering disabled

Monitor logging: level debugging, 0 messages logged, xml disabled,

filtering disabled

Buffer logging: level debugging, 92 messages logged, xml disabled,

filtering disabled

Exception Logging: size (4096 bytes)

Count and timestamp logging messages: disabled

Trap logging: level informational, 110 message lines logged

Logging to 10.216.16.60, 110 message lines logged, xml disabled,

filtering disabled

--More--

Log Buffer (16384 bytes):

00:01:35: curr is 0x0

00:01:35: RP: Currently running ROMMON from S (Gold) region

*Jan 9 03:14:14 IST: %SYS-6-CLOCKUPDATE: System clock has been updated from 21:44:14 UTC Fri Jan 8 2010 to 03:14:14 IST Sat Jan 9 2010, configured from console by console.

*Jan 9 03:14:17 IST: %SYS-5-CONFIG_I: Configured from memory by console

*Jan 9 03:14:20 IST: %SYS-5-RESTART: System restarted --

Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXH4, RELEASE SOFTWARE (fc1)

Technical Support:

Compiled Mon 10-Nov-08 07:00 by prod_rel_team

*Jan 9 03:14:20 IST: %NTP-6-RESTART: NTP process starts

*Jan 9 03:14:20 IST: %SNMP-5-COLDSTART: SNMP agent on host FWSM-Switch-Primary is undergoing a cold start

00:00:08: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Jan 9 03:15:36 IST: %SNMP-5-MODULETRAP: Module 2 [Up] Trap

Jan 9 03:15:36 IST: %OIR-SP-6-INSCARD: Card inserted in slot 2, interfaces are now online

*Jan 9 03:15:39 IST: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks

Jan 9 03:15:45 IST: %DIAG-SP-6-DIAG_OK: Module 1: Passed Online Diagnostics

Jan 9 03:15:50 IST: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 6.

Jan 9 03:15:50 IST: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in slot 6 became standby

*Jan 9 03:15:56 IST: %SNMP-5-MODULETRAP: Module 1 [Up] Trap

Jan 9 03:15:56 IST: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online

--More-- Jan 9 03:16:02 IST: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...

Jan 9 03:16:03 IST: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online Diagnostics

*Jan 9 03:16:05 IST: %SNMP-5-MODULETRAP: Module 6 [Up] Trap

Jan 9 03:16:05 IST: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online

Jan 9 03:16:10 IST: %DIAG-SP-6-RUN_MINIMUM: Module 3: Running Minimal Diagnostics...

*Jan 9 03:16:13 IST: %SVCLC-5-FWVTPMODE: VTP mode is set to non-transparent

*Jan 9 03:16:13 IST: %MFIB_CONST_RP-6-REPLICATION_MODE_CHANGE: Replication Mode Change Detected. Current system replication mode is Ingress

*Jan 9 03:16:13 IST: %SNMP-5-MODULETRAP: Module 3 [Up] Trap

Jan 9 03:16:13 IST: %DIAG-SP-6-DIAG_OK: Module 3: Passed Online Diagnostics

00:02:48: %SYS-SPSTBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

00:03:15: SPSTBY: SP: Currently running ROMMON from S (Gold) region

00:03:20: %DIAG-SPSTBY-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...

00:03:28: %DIAG-SPSTBY-6-DIAG_OK: Module 6: Passed Online Diagnostics

00:03:48: %SYS-SPSTBY-5-RESTART: System restarted --

Cisco IOS Software, s72033_sp Software (s72033_sp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXH4, RELEASE SOFTWARE (fc1)

Technical Support:

KARUPPUCHAMY MALAIYANDI · ‎01-12-2010

Hi

I am sorry...and we running 4.0(7).Please find the FWSM sh version details.....hope it helps u....

ITServerFW# sh version

FWSM Firewall Version 4.0(7)
Device Manager Version 6.0(3)

Compiled on Tue 08-Sep-09 20:48 by fwsmbld

ITServerFW up 115 days 14 hours
failover cluster up 301 days 0 hours

Hardware: WS-SVC-FWM-1, 1024 MB RAM, CPU Pentium III 1000 MHz
Flash STI Flash 8.0.0 @ 0xc321, 20MB

0: Int: GigabitEthernet0 : address is 0023.336a.dd00, irq 5
1: Int: GigabitEthernet1 : address is 0023.336a.dd00, irq 7
2: Int: EOBC0 : address is 0000.1100.0000, irq 11
The Running Activation Key is not set, using default settings:

Licensed features for this platform:
Maximum Interfaces : 256
Inside Hosts : Unlimited
Failover : Active/Active
VPN-DES : Enabled
VPN-3DES-AES : Enabled
Cut-through Proxy : Enabled
Guards : Enabled
URL Filtering : Enabled
Security Contexts : 2
GTP/GPRS : Disabled
BGP Stub : Disabled
Service Acceleration : Disabled
VPN Peers : Unlimited

View solution in original post

Panos Kampanakis · ‎01-12-2010

Madhu,

To answer your last question, you can upgrade from 3.1 to 4.0.

Make sure you have downtime though and if you have failover that the FWSMs are not running 3.1 and 4.0 at the same time.

I hope it helps.

PK

View solution in original post

KARUPPUCHAMY MALAIYANDI · ‎01-11-2010

Hi Madhu,

Even we faced the same issue on our one of the telecom customer in India.Whenever we will apply ACL , then CPU used to go around 95% and lots of latency.so we have decided do change the architecture.

If you have multiple L3 interfaces defined in FWSM that might be the one of the cause..because in this scenario the FWSM has to do routing and since all the L3 definition is in FWSM there will be lots of ARP entries.

In our scenario we have contacted cisco TAC, they suggest to one option is to remove all the L3 defintion in FWSM and change to SW ( restructuring the network) and another way is that upgrade the IOS into 4.1.

We have selected the first option .Now no more performance issue with FWSM.

Later ACL is reached the 10K, then we have upgraded IOS into 4.1 version and we have enabled ACL optimization.Now everything is going fine....

Hope it Helps..

Karuppuchamy CCIE(R&S),CCSP

madhusudhan s · ‎01-12-2010

Hi Karuppuchamy,

I could not find the sw version 4.1, only 4.0(9) is the latest version available. if you can send me the link it will be very helpfull for me.

Thanks & regards

Madhu

KARUPPUCHAMY MALAIYANDI · ‎01-12-2010

Hi

I am sorry...and we running 4.0(7).Please find the FWSM sh version details.....hope it helps u....

ITServerFW# sh version

FWSM Firewall Version 4.0(7)
Device Manager Version 6.0(3)

Compiled on Tue 08-Sep-09 20:48 by fwsmbld

ITServerFW up 115 days 14 hours
failover cluster up 301 days 0 hours

Hardware: WS-SVC-FWM-1, 1024 MB RAM, CPU Pentium III 1000 MHz
Flash STI Flash 8.0.0 @ 0xc321, 20MB

0: Int: GigabitEthernet0 : address is 0023.336a.dd00, irq 5
1: Int: GigabitEthernet1 : address is 0023.336a.dd00, irq 7
2: Int: EOBC0 : address is 0000.1100.0000, irq 11
The Running Activation Key is not set, using default settings:

Licensed features for this platform:
Maximum Interfaces : 256
Inside Hosts : Unlimited
Failover : Active/Active
VPN-DES : Enabled
VPN-3DES-AES : Enabled
Cut-through Proxy : Enabled
Guards : Enabled
URL Filtering : Enabled
Security Contexts : 2
GTP/GPRS : Disabled
BGP Stub : Disabled
Service Acceleration : Disabled
VPN Peers : Unlimited

madhusudhan s · ‎01-12-2010

Hi,

thanks for the suggestion, since you worked with TAC in this issue, I have few queries i would like to ask.

What all thing we can check to confirm that the latancy and the high utilization is due to the L3 defination in FWSM.

What is the back plane capacity for FWSM? i thing not sure but 60GB, isn't it enough to take the load(arp request and routing etc) ?

I need to justify my suggestion to our customer, Please help.

Regards

Madhu

madhusudhan s · ‎01-12-2010

Hi,

I am planning to upgrade the image from 3.1(10) to 4.0.(8), My question is whether i can directly upgrade from 3.1(10) to 4.0(8) or i need to upgrade to 4.0 then 4.0(8).

Regards

Madhu

Panos Kampanakis · ‎01-12-2010

Madhu,

To answer your last question, you can upgrade from 3.1 to 4.0.

Make sure you have downtime though and if you have failover that the FWSMs are not running 3.1 and 4.0 at the same time.

I hope it helps.

PK

madhusudhan s · ‎01-12-2010

Hi,

Thank you guys for the help..

I have sceduled for upgrading the FWSM by this weekend hope this will hepl in resolving the performance issue and other memory related issue which is faced.

Regards

Madhu

Carlo Poggiarelli · ‎01-20-2010

Hi Madhu,

Did the upgrade solve your issue?

We are facing a similar problem with our FWSM 4.0(4) and we are planning an upgrade to 4.0(9) to see if this will mitigate our issue.

Let me know

Bye

Carlo

Please rate all helpful posts "The more you help the more you learn"

madhusudhan s · ‎01-22-2010

Hi,

Yes it has helped me as i have upgraded from 3.1.10 to 4.0.8, so there are some ACL inhancement in new version when comparing with 3.x. Not sure how effective it would be for you as you will be upgrading the same(4.0.4 to 4.0.9) , may be some bug might have fixed in the latest.

regards

Madhu