05-13-2010 07:49 PM - edited 03-11-2019 10:45 AM
greetings
I have a pix 515 been running for years. last two days under some load the PIX just reboots
ive done
sh xlate count
sh conn count
sh mem
turned logging on for critical only
runs fine then,...whomp. it just reloads, like it dumps it's memory and reload it's config.
i have no idea how to read a crash dump. I thought maybe one you coulkd give me some insight on how to track down what is crashing on this pix.
thanks for any insight you may have.
here are 2 crash dumps i managed to copy;
05-15-2010 06:31 AM
HI,
Do you think you need to update the version of IOS a new version?
Version 6.3 (5) has several system errors.
Notes:
interface ethernet0 "outside" is up, line protocol is up
1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort
4387610 packets output, 753355575 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/128)
output queue (curr/max blocks): hardware (0/80) software (0/1)
interface ethernet1 "inside" is up, line protocol is up
input queue (curr/max blocks): hardware (128/128) software (0/80)
output queue (curr/max blocks): hardware (0/128) software (0/1)
interface ethernet2 "dmz" is up, line protocol is up
input queue (curr/max blocks): hardware (128/128) software (0/2)
output queue (curr/max blocks): hardware (0/1) software (0/1)
The hardware is full, the traffic of the network is using fairly saturated so the interface and produces overrun
send me the following show:
show cpu usage
show process (2 capture with a minute's difference between them)
show memory
show xlate count
show block
show local-host | in host| count/limit (place it as seen here)
show ssh sessions
Thanks and hope to be of Help You
05-17-2010 05:57 AM
CPU utilization for 5 seconds = 12%; 1 minute: 14%; 5 minutes: 17%
pixfirewall# sh cpu usage
CPU utilization for 5 seconds = 17%; 1 minute: 16%; 5 minutes: 15%
pixfirewall# show memory
Free memory: 46527928 bytes
Used memory: 20580936 bytes
------------- ----------------
Total memory: 67108864 bytes
pixfirewall# show xlate count
1092 in use, 1110 most used
pixfirewall# show block
SIZE MAX LOW CNT
4 1600 1594 1600
80 400 393 399
256 1012 963 1012
1550 1317 804 930
pixfirewall# show ssh sessions
pixfirewall# show process
PC SP STATE Runtime SBASE Stack Process
Hsi 001f02c9 0096f5fc 0056ed50 0 0096e674 3628/4096 arp_timer
Lsi 001f5a95 00a127f4 0056ed50 0 00a1187c 3816/4096 FragDBGC
Lwe 0011a13f 00a1e99c 005724b8 0 00a1db34 3688/4096 dbgtrace
Lwe 003fb2fd 00a20b2c 00567688 6930 00a1ebe4 6832/8192 Logger
Hsi 003ff455 00a23c24 0056ed50 0 00a21cac 8024/8192 tcp_fast
Hsi 003ff2f5 00a25cd4 0056ed50 0 00a23d5c 8024/8192 tcp_slow
Lsi 00314885 00b5c454 0056ed50 0 00b5b4cc 3916/4096 xlate clean
Lsi 00314793 00b5d4f4 0056ed50 0 00b5c57c 3884/4096 uxlate clean
Mwe 0030be5f 00cfd8f4 0056ed50 0 00cfb95c 7908/8192 tcp_intercept_timer_process
Lsi 00452ee5 00daa2cc 0056ed50 0 00da9344 3900/4096 route_process
Hsi 002fb6fc 00dab35c 0056ed50 0 00daa3f4 2732/4096 PIX Garbage Collector
Hwe 0021e529 00db588c 0056ed50 0 00db1924 16048/16384 isakmp_time_keeper
Lsi 002f929c 00dcf1cc 0056ed50 0 00dce244 3944/4096 perfmon
Mwe 00214d39 00df95fc 0056ed50 0 00df7684 7860/8192 IPsec timer handler
Hwe 003b105b 00e0da0c 00591c90 0 00e0bac4 7000/8192 qos_metric_daemon
Mwe 0026d0dd 00e28564 0056ed50 0 00e245fc 15592/16384 IP Background
Lwe 0030cad6 00edb274 00585368 0 00eda3fc 3704/4096 pix/trace
Lwe 0030cd0e 00edc324 00585a98 0 00edb4ac 3704/4096 pix/tconsole
Hwe 0011fa67 00ee81fc 0051bc10 0 00ee4814 14508/16384 ci/console
Hwe 0044c34a 00eea02c 005ebf78 0 00ee90f4 3684/4096 lu_ctl
Csi 003048fb 00eeb0fc 0056ed50 0 00eea1a4 3540/4096 update_cpu_usage
Hwe 002ef791 00f9bea4 0054e100 0 00f9801c 15884/16384 uauth_in
Hwe 003fdf05 00f9dfa4 008e6ac0 0 00f9c0cc 7896/8192 uauth_thread
Hwe 0041553a 00f9f0f4 00567c88 0 00f9e17c 3960/4096 udp_timer
Hsi 001e7d4e 00fa0db4 0056ed50 0 00f9fe3c 3928/4096 557mcfix
Crd 001e7d03 00fa1e74 0056f1c8 1325650 00fa0eec 3580/4096 557poll
Lsi 001e7dbd 00fa2f14 0056ed50 0 00fa1f9c 3848/4096 557timer
Cwe 001e99a9 00fb8fec 0056f1c8 96990 00fb70f4 6360/8192 pix/intf0
Mwe 004152aa 00fba0fc 00930cb0 0 00fb91c4 3896/4096 riprx/0
Msi 003ba8a1 00fbb20c 0056ed50 0 00fba294 3888/4096 riptx/0
Cwe 001e99a9 00fc1414 007ce078 102720 00fbf51c 6432/8192 pix/intf1
Mwe 004152aa 00fc2524 00930c68 0 00fc15ec 3896/4096 riprx/1
Msi 003ba8a1 00fc3634 0056ed50 0 00fc26bc 3888/4096 riptx/1
Cwe 001e99a9 00fc983c 00845618 100 00fc7944 7676/8192 pix/intf2
Mwe 004152aa 00fca94c 00930c20 0 00fc9a14 3896/4096 riprx/2
Msi 003ba8a1 00fcba5c 0056ed50 0 00fcaae4 3888/4096 riptx/2
Hwe 001b6151 00fccc24 005769d0 0 00fcbcbc 2844/4096 fover_thread
Hsi 0044d191 00fce3a4 0056ed50 0 00fcd42c 3928/4096 lu_xmit_timer
Hwe 0044be75 00fcf444 0056b438 0 00fce4dc 3900/4096 lu_rx
Hwe 0011fa67 0100b294 0051bc68 0 0100a5ec 3204/4096 fover_rx
Hwe 001b8f91 0100c614 00577064 0 0100b69c 3960/4096 fover_tx
Hwe 001b638c 0100e6c4 00577070 0 0100c74c 8056/8192 fover_rep
Lwe 001b6549 0100f784 00577078 0 0100e7fc 3976/4096 fover_lu_rep
Hwe 001b95e2 01013824 00577080 0 0100f8ac 16212/16384 fover_parse
Hwe 004152aa 01088aa4 00930b90 660 010880fc 848/4096 snmp
Hwe 004152aa 010896dc 00930bd8 0 01089394 812/1024 snmp_ex
Hwe 003fe199 0108a87c 008bd340 10 0108a234 1196/2048 listen/telnet_1
Mwe 0038707e 0108ca5c 0056ed50 0 0108aae4 7960/8192 Crypto CA
H* 003feba7 0009ff2c 0056ed38 6610 0134e55c 3908/8192 telnet/ci
pixfirewall# show process
PC SP STATE Runtime SBASE Stack Process
Hsi 001f02c9 0096f5fc 0056ed50 0 0096e674 3628/4096 arp_timer
Lsi 001f5a95 00a127f4 0056ed50 0 00a1187c 3816/4096 FragDBGC
Lwe 0011a13f 00a1e99c 005724b8 0 00a1db34 3688/4096 dbgtrace
Lwe 003fb2fd 00a20b2c 00567688 7400 00a1ebe4 6832/8192 Logger
Hsi 003ff455 00a23c24 0056ed50 0 00a21cac 8024/8192 tcp_fast
Hsi 003ff2f5 00a25cd4 0056ed50 0 00a23d5c 8024/8192 tcp_slow
Lsi 00314885 00b5c454 0056ed50 0 00b5b4cc 3916/4096 xlate clean
Lsi 00314793 00b5d4f4 0056ed50 0 00b5c57c 3884/4096 uxlate clean
Mwe 0030be5f 00cfd8f4 0056ed50 0 00cfb95c 7908/8192 tcp_intercept_timer_process
Lsi 00452ee5 00daa2cc 0056ed50 0 00da9344 3900/4096 route_process
Hsi 002fb6fc 00dab35c 0056ed50 0 00daa3f4 2732/4096 PIX Garbage Collector
Hwe 0021e529 00db588c 0056ed50 0 00db1924 16048/16384 isakmp_time_keeper
Lsi 002f929c 00dcf1cc 0056ed50 0 00dce244 3944/4096 perfmon
Mwe 00214d39 00df95fc 0056ed50 0 00df7684 7860/8192 IPsec timer handler
Hwe 003b105b 00e0da0c 00591c90 0 00e0bac4 7000/8192 qos_metric_daemon
Mwe 0026d0dd 00e28564 0056ed50 0 00e245fc 15592/16384 IP Background
Lwe 0030cad6 00edb274 00585368 0 00eda3fc 3704/4096 pix/trace
Lwe 0030cd0e 00edc324 00585a98 0 00edb4ac 3704/4096 pix/tconsole
Hwe 0011fa67 00ee81fc 0051bc10 0 00ee4814 14508/16384 ci/console
Hwe 0044c34a 00eea02c 005ebf78 0 00ee90f4 3684/4096 lu_ctl
Csi 003048fb 00eeb0fc 0056ed50 0 00eea1a4 3540/4096 update_cpu_usage
Hwe 002ef791 00f9bea4 0054e100 0 00f9801c 15884/16384 uauth_in
Hwe 003fdf05 00f9dfa4 008e6ac0 0 00f9c0cc 7896/8192 uauth_thread
Hwe 0041553a 00f9f0f4 00567c88 0 00f9e17c 3960/4096 udp_timer
Hsi 001e7d4e 00fa0db4 0056ed50 0 00f9fe3c 3928/4096 557mcfix
Crd 001e7d03 00fa1e74 0056f1c8 1385010 00fa0eec 3580/4096 557poll
Lsi 001e7dbd 00fa2f14 0056ed50 0 00fa1f9c 3848/4096 557timer
Cwe 001e99a9 00fb8fec 00756ae0 103300 00fb70f4 6360/8192 pix/intf0
Mwe 004152aa 00fba0fc 00930cb0 0 00fb91c4 3896/4096 riprx/0
Msi 003ba8a1 00fbb20c 0056ed50 0 00fba294 3888/4096 riptx/0
Cwe 001e99a9 00fc1414 007ce078 109140 00fbf51c 6256/8192 pix/intf1
Mwe 004152aa 00fc2524 00930c68 0 00fc15ec 3896/4096 riprx/1
Msi 003ba8a1 00fc3634 0056ed50 0 00fc26bc 3888/4096 riptx/1
Cwe 001e99a9 00fc983c 00845618 100 00fc7944 7676/8192 pix/intf2
Mwe 004152aa 00fca94c 00930c20 0 00fc9a14 3896/4096 riprx/2
Msi 003ba8a1 00fcba5c 0056ed50 0 00fcaae4 3888/4096 riptx/2
Hwe 001b6151 00fccc24 005769d0 0 00fcbcbc 2844/4096 fover_thread
Hsi 0044d191 00fce3a4 0056ed50 0 00fcd42c 3928/4096 lu_xmit_timer
Hwe 0044be75 00fcf444 0056b438 0 00fce4dc 3900/4096 lu_rx
Hwe 0011fa67 0100b294 0051bc68 0 0100a5ec 3204/4096 fover_rx
Hwe 001b8f91 0100c614 00577064 0 0100b69c 3960/4096 fover_tx
Hwe 001b638c 0100e6c4 00577070 0 0100c74c 8056/8192 fover_rep
Lwe 001b6549 0100f784 00577078 0 0100e7fc 3976/4096 fover_lu_rep
Hwe 001b95e2 01013824 00577080 0 0100f8ac 16212/16384 fover_parse
Hwe 004152aa 01088aa4 00930b90 660 010880fc 848/4096 snmp
Hwe 004152aa 010896dc 00930bd8 0 01089394 812/1024 snmp_ex
Hwe 003fe199 0108a87c 008bd340 10 0108a234 1196/2048 listen/telnet_1
Mwe 0038707e 0108ca5c 0056ed50 0 0108aae4 7960/8192 Crypto CA
H* 003feba7 0009ff2c 0056ed38 6630 0134e55c 3908/8192 telnet/ci
05-15-2010 06:48 AM
Please carefully reviewed this, I think that is the problem you have on the PIX
http://www.cisco.com/en/US/partner/ts/fn/100/fn15369.html
http://www.cisco.com/en/US/partner/ts/fn/100/fn15490.html
and this from Cisco:
Interface outside - ethernet0 (up/up) WARNING: There have been 1 'overruns' reported. This shows the number of times that the receiver hardware was incapable of handling received data to a hardware buffer because the input rate exceeded the receiver's capability to handle the data. If the overruns are equal to input errors and there are no CRC errors then at one point the ASA/PIX received packets faster than it can handle. This is not a cause of concern and can be ignored. TRY THIS: Verify that speed and duplex settings are hard-coded on the ASA/PIX and on the other directly connected devices. Use show blocks ASA/PIX command. A zero in the LOW column indicates a previous event where memory exhausted. A zero in the CNT column means memory is exhausted now. If the memory is continuously exhausted and traffic is not moving, then consider upgrading the interface to Gigabit or the ASA/PIX to a higher model. If this is DMZ interface, you can use other unused interfaces by splitting your current DMZ into 2 networks. If very large object-groups or large access-lists are used on ASA/PIX then use object-group-search keyword in the access-list ASA/PIX command to specify that access-list search is performed on object groups that are contained in access-list instead of searching the entire expanded access-list. Interface inside - ethernet1 (up/up) No Significant Errors to report. Interface dmz - ethernet2 (up/up) WARNING: More than 20% of packets received on this interface have been broadcasts. TRY THIS: Ensure that this level of broadcasts is required on this interface.
05-16-2010 05:56 PM
I can't access the two URLS you sent.
tomorrow during load I will capture those settings and post them.
when reading show blocks, im not seeing any zero's,... Ill investigate further.
i did find out that I am having some additional traffic hitting, I am NAT'ing 200 ip's and an additional 400 PAT's in addition to my regular traffic. these devices must be pushing this pIX to the edge. Although the pix should be able to handle this load,.. something causing a heap/reboot
more to come.
05-17-2010 06:48 AM
:: latest crash ::
5-17-10 9:26est
pixfirewall# sh crash
: Saved_Crash
Thread Name: PIX Garbage Collector (Old pc 0x002fb6fc ebp 0x00dab378)
Traceback:
0: 002f3ed7
1: 002f15d5
2: 002fb447
3: 002fb6b8
4: 00103b6d
5: 00000000
vector 0x0000000e (page fault)
edi 0x00000001
esi 0x00000005
ebp 0x00dab2f8
esp 0x00dab2bc
ebx 0x010bcf98
edx 0x010bcf98
ecx 0x00000000
eax 0x00000000
error code 0x00000000
eip 0x002f2265
cs 0x00000008
eflags 0x123456cd
CR2 0x0000004e
Stack dump: base:0x00daa3f4 size:4096, active:440
0x00dab3f0: 0x00000000
0x00dab3ec: 0x002fb4a8
0x00dab3e8-0x00dab3e4: 0x00000000
0x00dab3e0-0x00dab3c4: 0x12345678
0x00dab3c0: 0x00000000
0x00dab3bc: 0x00103b6d
0x00dab3b8: 0x00dab3e4
0x00dab3b4: 0x0000000a
0x00dab3b0-0x00dab39c: 0x12345678
0x00dab398-0x00dab390: 0x00000000
0x00dab38c: 0x002fb709
0x00dab388: 0x003f940e
0x00dab384: 0x010bcf98
0x00dab380: 0x00000000
0x00dab37c: 0x002fb6b8
0x00dab378: 0x00dab3b8
0x00dab374: 0x00fa304c
0x00dab370: 0x00faaf74
0x00dab36c: 0x0056ed50
0x00dab368: 0x00000000
0x00dab364: 0x003f940e
0x00dab360: 0x010bcf98
0x00dab35c: 0x002fb447
0x00dab358: 0x00dab378
0x00dab354: 0x00000246
0x00dab350: 0x00010030
0x00dab34c: 0x004de73e
0x00dab348: 0x00000033
0x00dab344: 0x00fa304c
0x00dab340: 0x00faaf8c
0x00dab33c: 0x00000246
0x00dab338: 0x00000002
0x00dab334: 0x00000005
0x00dab330: 0x010bcf98
0x00dab32c: 0x002f15d5
0x00dab328: 0x00dab358
0x00dab324: 0x00000000
0x00dab320: 0x003f1588
0x00dab31c: 0x00000000
0x00dab318: 0x003f9012
0x00dab314: 0x00fa304c
0x00dab310: 0x00000005
0x00dab30c: 0x12345678
0x00dab308: 0x00000005
0x00dab304: 0x010bcf98
0x00dab300: 0x00000000
0x00dab2fc: 0x002f3ed7
0x00dab2f8: 0x00dab328
0x00dab2f4: 0x00fa3038
0x00dab2f0: 0x013604c8
0x00dab2ec: 0x002f20a8
0x00dab2e8-0x00dab2e4: 0x00000001
0x00dab2e0: 0x00000005
0x00dab2dc: 0x00000516
0x00dab2d8: 0x186a60d1
0x00dab2d4: 0x00000005
0x00dab2d0: 0x00000000
0x00dab2cc: 0x002f3c99
0x00dab2c8: 0x00010207
0x00dab2c4: 0x00000008
0x00dab2c0: 0x002f2265
0x00dab2bc-0x00dab2b4: 0x00000000 *
0x00dab2b0-0x00dab2ac: 0x010bcf98
0x00dab2a8: 0x00dab2bc
0x00dab2a4: 0x00dab2f8
0x00dab2a0: 0x00000005
0x00dab29c: 0x00000001
0x00dab298: 0x0000000e
0x00dab294: 0x00105369
0x00dab290: 0x00dab2f8
0x00dab28c: 0x00314637
0x00dab288: 0x00000001
0x00dab284: 0x00000005
0x00dab280: 0x0117c258
0x00dab27c-0x00dab254: 0x00000000
0x00dab250: 0x00000001
0x00dab24c: 0x00000005
0x00dab248: 0x010bcf98
0x00dab244: 0xa813cac0
0x00dab240: 0x00000000
0x00dab23c: 0x85b00000
Cisco PIX Firewall Version 6.3(5)
The flash device is in use by another task.
Compiled on Thu 04-Aug-05 21:40 by morlee
pixfirewall up 1 hour 9 mins
Hardware: PIX-515, 64 MB RAM, CPU Pentium 200 MHz
Flash i28F640J5 @ 0x300, 16MB
BIOS Flash AT29C257 @ 0xfffd8000, 32KB
0: ethernet0: address is 0003.6bf6.69d8, irq 11
1: ethernet1: address is 0003.6bf6.69d9, irq 10
2: ethernet2: address is 0002.b349.5bf5, irq 7
Licensed Features:
Failover: Enabled
VPN-DES: Disabled
VPN-3DES-AES: Disabled
Maximum Physical Interfaces: 6
Maximum Interfaces: 10
Cut-through Proxy: Enabled
Guards: Enabled
URL-filtering: Enabled
Inside Hosts: Unlimited
Throughput: Unlimited
IKE peers: Unlimited
This PIX has an Unrestricted (UR) license.
Serial Number: 405250587 (0x1827a21b)
Running Activation Key: 0x3dcc8680 0x13297ae5 0x1a79c3f5 0x4a16039b
Configuration has not been modified since last system restart.
------------------ show clock ------------------
00:36:55.872 UTC Mon May 17 2010
------------------ show memory ------------------
Free memory: 46447328 bytes
Used memory: 20661536 bytes
------------- ----------------
Total memory: 67108864 bytes
------------------ show conn count ------------------
10954 in use, 11692 most used
------------------ show xlate count ------------------
1649 in use, 1651 most used
------------------ show blocks ------------------
SIZE MAX LOW CNT
4 1600 1594 1600
80 400 393 400
256 1012 963 1012
1550 1317 754 932
------------------ show interface ------------------
interface ethernet0 "outside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0003.6bf6.69d8
IP address 192.60.1.2, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
6945537 packets input, 3653371959 bytes, 0 no buffer
Received 15245 broadcasts, 0 runts, 0 giants
2 input errors, 0 CRC, 0 frame, 2 overrun, 0 ignored, 0 abort
5676618 packets output, 770938923 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/128)
output queue (curr/max blocks): hardware (0/73) software (0/1)
interface ethernet1 "inside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0003.6bf6.69d9
IP address 192.40.1.2, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
5692470 packets input, 772166970 bytes, 0 no buffer
Received 111 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
6978878 packets output, 3662006607 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/73)
output queue (curr/max blocks): hardware (0/84) software (0/1)
interface ethernet2 "dmz" is up, line protocol is up
Hardware is i82559 ethernet, address is 0002.b349.5bf5
IP address 127.0.0.1, subnet mask 255.255.255.255
MTU 1500 bytes, BW 100000 Kbit full duplex
5938 packets input, 356100 bytes, 0 no buffer
Received 5935 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
7 packets output, 420 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/2)
output queue (curr/max blocks): hardware (0/1) software (0/1)
------------------ show cpu usage ------------------
CPU utilization for 5 seconds = 34%; 1 minute: 24%; 5 minutes: 22%
------------------ show process ------------------
PC SP STATE Runtime SBASE Stack Process
Hsi 001f02c9 0096f5fc 0056ed50 0 0096e674 3628/4096 arp_timer
Lsi 001f5a95 00a127f4 0056ed50 0 00a1187c 3816/4096 FragDBGC
Lwe 0011a13f 00a1e99c 005724b8 0 00a1db34 3688/4096 dbgtrace
Lwe 003fb2fd 00a20b2c 00567688 15760 00a1ebe4 6832/8192 Logger
Hrd 003ff455 00a23c24 0056ed38 0 00a21cac 8024/8192 tcp_fast
Hrd 003ff2f5 00a25cd4 0056ed38 0 00a23d5c 8024/8192 tcp_slow
Lsi 00314885 00b5c454 0056ed50 0 00b5b4cc 3916/4096 xlate clean
Lsi 00314793 00b5d4f4 0056ed50 0 00b5c57c 3884/4096 uxlate clean
Mwe 0030be5f 00cfd8f4 0056ed50 0 00cfb95c 7908/8192 tcp_intercept_timer_process
Lsi 00452ee5 00daa2cc 0056ed50 0 00da9344 3900/4096 route_process
H* 002fb6fc 0009ff2c 0056ed38 0 00daa3f4 1780/4096 PIX Garbage Collector
Hwe 0021e529 00db588c 0056ed50 0 00db1924 16048/16384 isakmp_time_keeper
Lsi 002f929c 00dcf1cc 0056ed50 0 00dce244 3944/4096 perfmon
Mwe 00214d39 00df95fc 0056ed50 0 00df7684 7860/8192 IPsec timer handler
Hwe 003b105b 00e0da0c 00591c90 0 00e0bac4 7000/8192 qos_metric_daemon
Mwe 0026d0dd 00e28564 0056ed50 10 00e245fc 15592/16384 IP Background
Lwe 0030cad6 00edb274 00585368 0 00eda3fc 3704/4096 pix/trace
Lwe 0030cd0e 00edc324 00585a98 0 00edb4ac 3704/4096 pix/tconsole
Hwe 0011fa67 00ee81fc 0051bc10 0 00ee4814 14508/16384 ci/console
Hwe 0044c34a 00eea02c 005ebf78 0 00ee90f4 3684/4096 lu_ctl
Csi 003048fb 00eeb0fc 0056ed50 0 00eea1a4 3540/4096 update_cpu_usage
Hwe 002ef791 00f9bea4 0054e100 0 00f9801c 15884/16384 uauth_in
Hwe 003fdf05 00f9dfa4 008e6ac0 0 00f9c0cc 7896/8192 uauth_thread
Hwe 0041553a 00f9f0f4 00567c88 0 00f9e17c 3960/4096 udp_timer
Hsi 001e7d4e 00fa0db4 0056ed50 0 00f9fe3c 3928/4096 557mcfix
Crd 001e7d03 00fa1e74 0056f1c8 2464110 00fa0eec 3580/4096 557poll
Lsi 001e7dbd 00fa2f14 0056ed50 0 00fa1f9c 3848/4096 557timer
Cwe 001e99a9 00fb8fec 00756ae0 199220 00fb70f4 6360/8192 pix/intf0
Mwe 004152aa 00fba0fc 00930cb0 0 00fb91c4 3896/4096 riprx/0
Msi 003ba8a1 00fbb20c 0056ed50 0 00fba294 3888/4096 riptx/0
Cwe 001e99a9 00fc1414 007ce078 220230 00fbf51c 6256/8192 pix/intf1
Mwe 004152aa 00fc2524 00930c68 0 00fc15ec 3896/4096 riprx/1
Msi 003ba8a1 00fc3634 0056ed50 0 00fc26bc 3888/4096 riptx/1
Cwe 001e99a9 00fc983c 00845618 170 00fc7944 7672/8192 pix/intf2
Mwe 004152aa 00fca94c 00930c20 0 00fc9a14 3896/4096 riprx/2
Msi 003ba8a1 00fcba5c 0056ed50 0 00fcaae4 3888/4096 riptx/2
Hwe 001b6151 00fccc24 005769d0 0 00fcbcbc 2844/4096 fover_thread
Hsi 0044d191 00fce3a4 0056ed50 0 00fcd42c 3928/4096 lu_xmit_timer
Hwe 0044be75 00fcf444 0056b438 0 00fce4dc 3900/4096 lu_rx
Hwe 0011fa67 0100b294 0051bc68 0 0100a5ec 3204/4096 fover_rx
Hwe 001b8f91 0100c614 00577064 0 0100b69c 3960/4096 fover_tx
Hwe 001b638c 0100e6c4 00577070 0 0100c74c 8056/8192 fover_rep
Lwe 001b6549 0100f784 00577078 0 0100e7fc 3976/4096 fover_lu_rep
Hwe 001b95e2 01013824 00577080 0 0100f8ac 16212/16384 fover_parse
Hwe 004152aa 01088aa4 00930b90 1060 010880fc 800/4096 snmp
Hwe 004152aa 010896dc 00930bd8 0 01089394 812/1024 snmp_ex
Hwe 003fe199 0108a87c 008bd340 10 0108a234 1196/2048 listen/telnet_1
Mwe 0038707e 0108ca5c 0056ed50 0 0108aae4 7960/8192 Crypto CA
Hwe 003feba7 0134fe94 008bd248 6630 0134e55c 3908/8192 telnet/ci
------------------ show failover ------------------
Failover Off
Cable status: My side not connected
Reconnect timeout 0:00:00
Poll frequency 15 seconds
------------------ show traffic ------------------
outside:
received (in 4166.820 secs):
6945537 packets 3653371959 bytes
1666 pkts/sec 876777 bytes/sec
transmitted (in 4166.820 secs):
5676618 packets 770938923 bytes
1362 pkts/sec 185018 bytes/sec
inside:
received (in 4166.820 secs):
5692470 packets 772166970 bytes
1366 pkts/sec 185313 bytes/sec
transmitted (in 4166.820 secs):
6978878 packets 3662006607 bytes
1674 pkts/sec 878849 bytes/sec
dmz:
received (in 4166.820 secs):
5938 packets 356100 bytes
1 pkts/sec 85 bytes/sec
transmitted (in 4166.820 secs):
7 packets 420 bytes
0 pkts/sec 0 bytes/sec
------------------ show perfmon ------------------
PERFMON STATS: Current Average
Xlates 3/s 1/s
Connections 127/s 117/s
TCP Conns 124/s 114/s
UDP Conns 2/s 3/s
URL Access 26/s 25/s
URL Server Req 0/s 0/s
TCP Fixup 5195/s 934/s
TCPIntercept 0/s 0/s
HTTP Fixup 4371/s 166/s
FTP Fixup 0/s 0/s
AAA Authen 0/s 0/s
AAA Author 0/s 0/s
AAA Account 0/s 0/s
: End_Crash
pixfirewall#
05-17-2010 07:18 AM
and another
Last login: Mon May 17 09:47:06 on ttys012
nc1-100:~ drfoo$ telnet 192.40.1.2
Trying 192.40.1.2...
Connected to 192.40.1.2.
Escape character is '^]'.
User Access Verification
Password:
Type help or '?' for a list of available commands.
pixfirewall> en
Password:
pixfirewall# sh crash
: Saved_Crash
Thread Name: PIX Garbage Collector (Old pc 0x002fb6fc ebp 0x00dab378)
Traceback:
0: 0030c8cc
1: 002f3d03
2: 002f3ed7
3: 002f15d5
4: 002fb447
5: 002fb6b8
6: 00103b6d
7: 00000000
vector 0x0000000e (page fault)
edi 0x00000000
esi 0x00000005
ebp 0x00dab298
esp 0x00dab288
ebx 0x010bd6f0
edx 0x00db18b0
ecx 0x0006001d
eax 0x0006001d
error code 0x00000000
eip 0x001fe372
cs 0x00000008
eflags 0x123456cd
CR2 0x0006001d
Stack dump: base:0x00daa3f4 size:4096, active:492
0x00dab3f0: 0x00000000
0x00dab3ec: 0x002fb4a8
0x00dab3e8-0x00dab3e4: 0x00000000
0x00dab3e0-0x00dab3c4: 0x12345678
0x00dab3c0: 0x00000000
0x00dab3bc: 0x00103b6d
0x00dab3b8: 0x00dab3e4
0x00dab3b4: 0x0000000a
0x00dab3b0-0x00dab39c: 0x12345678
0x00dab398-0x00dab390: 0x00000000
0x00dab38c: 0x002fb709
0x00dab388: 0x000eac0e
0x00dab384: 0x010bd6f0
0x00dab380: 0x00000000
0x00dab37c: 0x002fb6b8
0x00dab378: 0x00dab3b8
0x00dab374: 0x00fa304c
0x00dab370: 0x00fa4d4c
0x00dab36c: 0x0056ed50
0x00dab368: 0x00000000
0x00dab364: 0x000eac0e
0x00dab360: 0x010bd6f0
0x00dab35c: 0x002fb447
0x00dab358: 0x00dab378
0x00dab354: 0x00000246
0x00dab350: 0x00010030
0x00dab34c: 0x004de73e
0x00dab348: 0x00000029
0x00dab344: 0x00fa304c
0x00dab340: 0x00fa4dbc
0x00dab33c: 0x00000246
0x00dab338: 0x00000002
0x00dab334: 0x00000005
0x00dab330: 0x010bd6f0
0x00dab32c: 0x002f15d5
0x00dab328: 0x00dab358
0x00dab324: 0x00000000
0x00dab320: 0x000c3a3c
0x00dab31c: 0x00000000
0x00dab318: 0x000ea40c
0x00dab314: 0x00000001
0x00dab310: 0x00000005
0x00dab30c: 0x12345678
0x00dab308: 0x00000005
0x00dab304: 0x010bd6f0
0x00dab300: 0x00db18b0
0x00dab2fc: 0x002f3ed7
0x00dab2f8: 0x00dab328
0x00dab2f4: 0x00fa3038
0x00dab2f0: 0x01368c38
0x00dab2ec: 0x002f20a8
0x00dab2e8-0x00dab2e4: 0x00000001
0x00dab2e0: 0x00000005
0x00dab2dc: 0x00005f07
0x00dab2d8: 0x186a60d1
0x00dab2d4: 0x00000005
0x00dab2d0: 0x010bd6f0
0x00dab2cc: 0x002f3d03
0x00dab2c8: 0x00dab2f8
0x00dab2c4: 0x000000da
0x00dab2c0: 0x013469dc
0x00dab2bc: 0x075f0000
0x00dab2b8: 0x00000000
0x00dab2b4: 0x010a0390
0x00dab2b0: 0x010bd6f0
0x00dab2ac: 0x00000000
0x00dab2a8: 0x0000003f
0x00dab2a4: 0x010c17f0
0x00dab2a0: 0x0006001d
0x00dab29c: 0x0030c8cc
0x00dab298: 0x00dab2c8
0x00dab294: 0x00010206
0x00dab290: 0x00000008
0x00dab28c: 0x001fe372
0x00dab288: 0x00000000 *
0x00dab284-0x00dab280: 0x0006001d
0x00dab27c: 0x00db18b0
0x00dab278: 0x010bd6f0
0x00dab274: 0x00dab288
0x00dab270: 0x00dab298
0x00dab26c: 0x00000005
0x00dab268: 0x00000000
0x00dab264: 0x0000000e
0x00dab260: 0x00105369
0x00dab25c: 0x00dab298
0x00dab258: 0x00000000
0x00dab254: 0x010a0390
0x00dab250: 0x01113230
0x00dab24c: 0x00000070
0x00dab248: 0x0001db00
0x00dab244: 0xa813cac0
0x00dab240: 0x0001daba
0x00dab23c: 0x00000014
0x00dab238: 0x00dab2f8
0x00dab234: 0x17109711
0x00dab230: 0x00581178
0x00dab22c: 0x00025007
0x00dab228: 0x00000000
0x00dab224: 0x00000005
0x00dab220: 0x010bd6f0
0x00dab21c: 0x00000000
0x00dab218: 0x00000005
0x00dab214: 0x010bd6f0
0x00dab210: 0x002ec300
0x00dab20c: 0x00000000
0x00dab208: 0x85b00000
Cisco PIX Firewall Version 6.3(5)
The flash device is in use by another task.
Compiled on Thu 04-Aug-05 21:40 by morlee
pixfirewall up 16 mins 1 sec
Hardware: PIX-515, 64 MB RAM, CPU Pentium 200 MHz
Flash i28F640J5 @ 0x300, 16MB
BIOS Flash AT29C257 @ 0xfffd8000, 32KB
0: ethernet0: address is 0003.6bf6.69d8, irq 11
1: ethernet1: address is 0003.6bf6.69d9, irq 10
2: ethernet2: address is 0002.b349.5bf5, irq 7
Licensed Features:
Failover: Enabled
VPN-DES: Disabled
VPN-3DES-AES: Disabled
Maximum Physical Interfaces: 6
Maximum Interfaces: 10
Cut-through Proxy: Enabled
Guards: Enabled
URL-filtering: Enabled
Inside Hosts: Unlimited
Throughput: Unlimited
IKE peers: Unlimited
This PIX has an Unrestricted (UR) license.
Serial Number: 405250587 (0x1827a21b)
Running Activation Key: 0x3dcc8680 0x13297ae5 0x1a79c3f5 0x4a16039b
Configuration has not been modified since last system restart.
------------------ show clock ------------------
01:23:16.704 UTC Mon May 17 2010
------------------ show memory ------------------
Free memory: 46542488 bytes
Used memory: 20566376 bytes
------------- ----------------
Total memory: 67108864 bytes
------------------ show conn count ------------------
9294 in use, 11532 most used
------------------ show xlate count ------------------
1268 in use, 1277 most used
------------------ show blocks ------------------
SIZE MAX LOW CNT
4 1600 1593 1600
80 400 393 400
256 1012 958 1012
1550 1317 766 932
------------------ show interface ------------------
interface ethernet0 "outside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0003.6bf6.69d8
IP address 192.60.1.2, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
2146717 packets input, 2272654601 bytes, 0 no buffer
Received 2051 broadcasts, 0 runts, 0 giants
13 input errors, 0 CRC, 0 frame, 13 overrun, 0 ignored, 0 abort
1878299 packets output, 291981089 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/128)
output queue (curr/max blocks): hardware (0/33) software (0/1)
interface ethernet1 "inside" is up, line protocol is up
Hardware is i82559 ethernet, address is 0003.6bf6.69d9
IP address 192.40.1.2, subnet mask 255.255.255.0
MTU 1500 bytes, BW 100000 Kbit full duplex
1884832 packets input, 292605301 bytes, 0 no buffer
Received 26 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
2208007 packets output, 2283607215 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/33)
output queue (curr/max blocks): hardware (0/65) software (0/1)
interface ethernet2 "dmz" is up, line protocol is up
Hardware is i82559 ethernet, address is 0002.b349.5bf5
IP address 127.0.0.1, subnet mask 255.255.255.255
MTU 1500 bytes, BW 100000 Kbit full duplex
1359 packets input, 81540 bytes, 0 no buffer
Received 1359 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
7 packets output, 420 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (128/128) software (0/2)
output queue (curr/max blocks): hardware (0/1) software (0/1)
------------------ show cpu usage ------------------
CPU utilization for 5 seconds = 19%; 1 minute: 20%; 5 minutes: 22%
------------------ show process ------------------
PC SP STATE Runtime SBASE Stack Process
Hsi 001f02c9 0096f5fc 0056ed50 0 0096e674 3628/4096 arp_timer
Lsi 001f5a95 00a127f4 0056ed50 0 00a1187c 3928/4096 FragDBGC
Lwe 0011a13f 00a1e99c 005724b8 0 00a1db34 3688/4096 dbgtrace
Lwe 003fb2fd 00a20b2c 00567688 10480 00a1ebe4 6836/8192 Logger
Hwe 003ff4b8 00a23c24 00567938 0 00a21cac 8024/8192 tcp_fast
Hwe 003ff431 00a25cd4 00567938 0 00a23d5c 8024/8192 tcp_slow
Lsi 00314885 00b5c454 0056ed50 0 00b5b4cc 3916/4096 xlate clean
Lsi 00314793 00b5d4f4 0056ed50 0 00b5c57c 3884/4096 uxlate clean
Mwe 0030be5f 00cfd8f4 0056ed50 0 00cfb95c 7908/8192 tcp_intercept_timer_process
Lsi 00452ee5 00daa2cc 0056ed50 0 00da9344 3900/4096 route_process
H* 002fb6fc 0009ff2c 0056ed38 0 00daa3f4 1728/4096 PIX Garbage Collector
Hwe 0021e529 00db588c 0056ed50 0 00db1924 16048/16384 isakmp_time_keeper
Lsi 002f929c 00dcf1cc 0056ed50 0 00dce244 3944/4096 perfmon
Mwe 00214d39 00df95fc 0056ed50 0 00df7684 7860/8192 IPsec timer handler
Hwe 003b105b 00e0da0c 00591c90 0 00e0bac4 7000/8192 qos_metric_daemon
Mwe 0026d0dd 00e28564 0056ed50 0 00e245fc 15592/16384 IP Background
Lwe 0030cad6 00edb274 00585368 0 00eda3fc 3704/4096 pix/trace
Lwe 0030cd0e 00edc324 00585a98 0 00edb4ac 3704/4096 pix/tconsole
Hwe 0011fa67 00ee81fc 0051bc10 0 00ee4814 14508/16384 ci/console
Hwe 0044c34a 00eea02c 005ebf78 0 00ee90f4 3684/4096 lu_ctl
Csi 003048fb 00eeb0fc 0056ed50 0 00eea1a4 3540/4096 update_cpu_usage
Hwe 002ef791 00f9bea4 0054e100 0 00f9801c 15884/16384 uauth_in
Hwe 003fdf05 00f9dfa4 008e6ac0 0 00f9c0cc 7896/8192 uauth_thread
Hwe 0041553a 00f9f0f4 00567c88 0 00f9e17c 3960/4096 udp_timer
Hsi 001e7d4e 00fa0db4 0056ed50 0 00f9fe3c 3928/4096 557mcfix
Crd 001e7d03 00fa1e74 0056f1c8 531540 00fa0eec 3584/4096 557poll
Lsi 001e7dbd 00fa2f14 0056ed50 0 00fa1f9c 3848/4096 557timer
Cwe 001e99a9 00fb8fec 00756ae0 63360 00fb70f4 6256/8192 pix/intf0
Mwe 004152aa 00fba0fc 00930cb0 0 00fb91c4 3896/4096 riprx/0
Msi 003ba8a1 00fbb20c 0056ed50 0 00fba294 3888/4096 riptx/0
Cwe 001e99a9 00fc1414 007ce078 78310 00fbf51c 6360/8192 pix/intf1
Mwe 004152aa 00fc2524 00930c68 0 00fc15ec 3896/4096 riprx/1
Msi 003ba8a1 00fc3634 0056ed50 0 00fc26bc 3888/4096 riptx/1
Cwe 001e99a9 00fc983c 00845618 70 00fc7944 7676/8192 pix/intf2
Mwe 004152aa 00fca94c 00930c20 0 00fc9a14 3896/4096 riprx/2
Msi 003ba8a1 00fcba5c 0056ed50 0 00fcaae4 3888/4096 riptx/2
Hwe 001b6151 00fccc24 005769d0 0 00fcbcbc 2844/4096 fover_thread
Hrd 0044d191 00fce3a4 0056ed38 0 00fcd42c 3928/4096 lu_xmit_timer
Hwe 0044be75 00fcf444 0056b438 0 00fce4dc 3900/4096 lu_rx
Hwe 0011fa67 0100b294 0051bc68 0 0100a5ec 3204/4096 fover_rx
Hwe 001b8f91 0100c614 00577064 0 0100b69c 3960/4096 fover_tx
Hwe 001b638c 0100e6c4 00577070 0 0100c74c 8056/8192 fover_rep
Lwe 001b6549 0100f784 00577078 0 0100e7fc 3976/4096 fover_lu_rep
Hwe 001b95e2 01013824 00577080 0 0100f8ac 16212/16384 fover_parse
Hwe 004152aa 01088aa4 00930b90 220 010880fc 784/4096 snmp
Hwe 004152aa 010896dc 00930bd8 0 01089394 812/1024 snmp_ex
Hwe 003fe199 0108a87c 008bd248 0 0108a234 1196/2048 listen/telnet_1
Mwe 0038707e 0108ca5c 0056ed50 0 0108aae4 7960/8192 Crypto CA
------------------ show failover ------------------
Failover Off
Cable status: My side not connected
Reconnect timeout 0:00:00
Poll frequency 15 seconds
------------------ show traffic ------------------
outside:
received (in 961.690 secs):
2146717 packets 2272654601 bytes
2232 pkts/sec 2363188 bytes/sec
transmitted (in 961.690 secs):
1878299 packets 291981089 bytes
1953 pkts/sec 303612 bytes/sec
inside:
received (in 961.690 secs):
1884832 packets 292605301 bytes
1959 pkts/sec 304261 bytes/sec
transmitted (in 961.690 secs):
2208007 packets 2283607215 bytes
2295 pkts/sec 2374577 bytes/sec
dmz:
received (in 961.690 secs):
1359 packets 81540 bytes
1 pkts/sec 84 bytes/sec
transmitted (in 961.690 secs):
7 packets 420 bytes
0 pkts/sec 0 bytes/sec
------------------ show perfmon ------------------
PERFMON STATS: Current Average
Xlates 1/s 2/s
Connections 155/s 178/s
TCP Conns 152/s 174/s
UDP Conns 3/s 3/s
URL Access 67/s 79/s
URL Server Req 0/s 0/s
TCP Fixup 3949/s 4087/s
TCPIntercept 0/s 0/s
HTTP Fixup 2950/s 3169/s
FTP Fixup 0/s 0/s
AAA Authen 0/s 0/s
AAA Author 0/s 0/s
AAA Account 0/s 0/s
: End_Crash
pixfirewall#
05-18-2010 03:18 PM
05-18-2010 05:30 PM
yup, that must be it. funny that there isn't anything on the console or logs that jumps right out and says " hey something is filling up Im rebooting "
Im looking to replace the unit, I don't see my load decreasing.
I wonder if rate limit from specific ip ranges would help. your thoughts ?
-j
05-18-2010 07:21 PM
Yes, you can check the connections. These statistics can be viewed with the command:
show local-host | in host | count / limit
before I send you this command.
The log of this version is not as robust. The 7.x version has significant improvements. Have you thought about upgrading?
send me the output.
05-18-2010 07:46 PM
yes you sent me that command earlier, works great. I never new the pix could do regex, im in heaven.
upgrade to 7.x - it's a 515, it can't support 7 with it's current memory.
i have been looking on line, tons of 525's out there for less than $500.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide