Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

Buffer leak from SSLVPN

Hello,

I’m running a Cisco 3845 with a AIM-VPN/SSL-3 Module. My WAN Interface (a DSL Connection) is configured with NAT overload. This is also the interface where the sslvpn connection is terminating. If a user connects from the internet to the SSLVPN in full tunnel mode and uses the internet a buffer leak occurs in the middle pool and after certain time the router crashes. The IOS which is running on the device is c3845-adventerprisek9-mz.151-4.M7.bin. I also tried out the latest Version c3845-adventerprisek9-mz.151-4.M8.bin but it doesn't make any differences.

sh buffers
Buffer elements:
     674 in free list (500 max allowed)
     3064635 hits, 0 misses, 617 created

Public buffer pools:
Small buffers, 104 bytes (total 183, permanent 150, peak 183 @ 01:03:44):
     176 in free list (50 min, 300 max allowed)
     959716 hits, 194 misses, 13 trims, 46 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 823, permanent 400, peak 823 @ 00:45:03):
     458 in free list (400 min, 800 max allowed)
     91468 hits, 225 misses, 69 trims, 492 created
     0 failures (0 no memory)
Big buffers, 1536 bytes (total 741, permanent 500, peak 773 @ 01:03:46):
     685 in free list (500 min, 1000 max allowed)
     1333349 hits, 519 misses, 75 trims, 316 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 45, permanent 45, peak 48 @ 04:16:10):
     44 in free list (35 min, 65 max allowed)
     183629 hits, 1 misses, 3 trims, 3 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 35, permanent 35, peak 36 @ 04:16:10):
     35 in free list (25 min, 65 max allowed)
     0 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 4, permanent 4, peak 5 @ 04:16:10):
     4 in free list (2 min, 8 max allowed)
     0 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)

The 'sh buffer' output above is quite fresh after a reload of the device. When users generate a lot of traffic to the internet from a Full tunnel Connection the used buffers in the middle pool rises until no IO memory is left on the device. I can check this with 'show memory statistic history' :

 

      555555555555555555555555555555555555555555555555555556666666
      999999999999999999999999999999999999999999999999999990000000
  100                                                             
   90                                                             
   80                                                             
   70                                                             
   60 ############################################################
   50 ############################################################
   40 ############################################################
   30 ############################################################
   20 ############################################################
   10 ############################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               Free memory per minute (last 60 minutes)
              * = maximum # = average

About 50mins ago 1% of the IO memory become wasted due to a leak. This was the time where I tested it with a Anyconnect client from my mobile device and opened one website.

 

Does anyone have an Idea how to fix this? Appreciated any kind of help.

 

kind regards

 

Lukasz

3 REPLIES
New Member

It looks like even if someone

It looks like even if someone tries to access resources from the LAN through a full tunnel a leak happens as well:

 

sh buffers
Buffer elements:
     674 in free list (500 max allowed)
     3359687 hits, 0 misses, 617 created

Public buffer pools:
Small buffers, 104 bytes (total 150, permanent 150, peak 183 @ 02:55:10):
     143 in free list (50 min, 300 max allowed)
     1051942 hits, 194 misses, 46 trims, 46 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 21962, permanent 400, peak 21962 @ 00:04:45):
     407 in free list (400 min, 800 max allowed)
     141634 hits, 15220 misses, 123 trims, 21685 created
     0 failures (0 no memory)
Big buffers, 1536 bytes (total 582, permanent 500, peak 773 @ 02:55:13):
     521 in free list (500 min, 1000 max allowed)
     1377048 hits, 543 misses, 260 trims, 342 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 45, permanent 45, peak 48 @ 06:07:36):
     44 in free list (35 min, 65 max allowed)
     184351 hits, 1 misses, 3 trims, 3 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 35, permanent 35, peak 36 @ 06:07:36):
     35 in free list (25 min, 65 max allowed)
     0 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 4, permanent 4, peak 5 @ 06:07:36):
     4 in free list (2 min, 8 max allowed)
     0 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)

IO Memory:

                                                                 

      1111111111111223334455566666666666666666666666666666655555555
      999999999999937159371590000000000000000000000000000009999999
  100                                                             
   90                                                             
   80                                                             
   70                                                             
   60                                            *######################################
   50                                        *########################################
   40                                  *###########################################
   30                              ##############################################
   20 ############################################################
   10 ############################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               Free memory per minute (last 60 minutes)
              * = maximum # = average

 

 

 

This happens after I downloaded a ~60MB file from a CIFS share through a full tunnel.

New Member

I continued using VPN and

I continued using VPN and downloaded more files from the CIFS server until all IO Memory was exhausted. The last messages from the router where theses:

186>1 2014-09-10T18:02:07.182019+02:00 hostname 190 - - -  Sep 10 16:02:06.166: %SYS-2-MALLOCFAIL: Memory allocation of 780 bytes failed from 0x606625D0, alignment 32
<186>1 2014-09-10T18:02:07.182019+02:00 hostname 191 - - -  Pool: I/O  Free: 16544  Cause: Memory fragmentation
<186>1 2014-09-10T18:02:07.182074+02:00 hostname 192 - - -  Alternate Pool: None  Free: 0  Cause: No Alternate pool
<186>1 2014-09-10T18:02:07.182074+02:00 hostname 193 - - -   -Process= "Pool Manager", ipl= 0, pid= 7
<186>1 2014-09-10T18:02:07.182089+02:00 hostname 194 - - -  -Traceback= 63F40FD8z 6065CD98z 63F5DEECz 63F5E268z 63F0D888z 63F0D86Cz
<186>1 2014-09-10T18:02:37.187832+02:00 hostname 195 - - -  Sep 10 16:02:36.183: %SYS-2-MALLOCFAIL: Memory allocation of 780 bytes failed from 0x606625D0, alignment 32
<186>1 2014-09-10T18:02:37.187832+02:00 hostname 196 - - -  Pool: I/O  Free: 16544  Cause: Memory fragmentation
<186>1 2014-09-10T18:02:37.187924+02:00 hostname 197 - - -  Alternate Pool: None  Free: 0  Cause: No Alternate pool
<186>1 2014-09-10T18:02:37.187924+02:00 hostname 198 - - -   -Process= "Pool Manager", ipl= 0, pid= 7
<186>1 2014-09-10T18:02:37.188421+02:00 hostname 199 - - -  -Traceback= 63F40FD8z 6065CD98z 63F5DEECz 63F5E178z 63F0D888z 63F0D86Cz
<185>1 2014-09-10T18:02:43.119863+02:00 hostname 200 - - -  Sep 10 16:02:43.111: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, UNKNOWN, handle=0x100DC

 

After that the ssh session got terminated and Internet access was barley possible. I had to restart the device.

New Member

Ok, it looks like sslvpn code

Ok, it looks like sslvpn code is broken in  IOS 15.1(4)M - see CSCug17485. However, the memory leak only occurs if clients connect from the internet to the NAT-Enabled WAN interface and generated bulk traffic. If I connect from the inside of the network to the webvpn Server (same Interface) and generate traffic everything is fine.

IOS 15.0 hasn't this issue but there I can't get windows7 clients to authenticate probably using the anyconnect client. First I got the error message "anyconnect cannot confirm it is connected to your secure gateway". After some research I tried to import the Certificate by hand and the next error message i got is "anyconnect connection attempt has failed due to network or pc issue". I deleted all tempfiles associated with anyconnect, checked firewall and antivirus, reinstalled the client and tried different version but no success. Anyway a connection from a android mobile device with the latest anyconnect client works fine.

Anybody an idea how to get this up & running on IOS 15.0-1.M10

 

kind regards

Lukasz

146
Views
5
Helpful
3
Replies