Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

Understanding and Troubleshooting IP Pools

Includes basics of IP pool configuration and troubleshooting issues related to IP pools, including examining output such as monitor subscriber, logs, SNMP, alarms, thresholds.

IP pools are ranges of IP addresses that are reserved to be doled out to subscribers when they connect to the network. The system can be designed to have all service types (MIP, SIP, LNS, etc) share a given pool, or pools can be unique per service type and subscriber grouping – the flexibility is there to allow for limitless possibilities from simple to complex. This article is not intended for planning and architecting IP pools and contexts for a new (or existing) system – that is best addressed by a Starent network consultant. Rather, this article explains how to view various aspects of ip pools, troubleshoot ip pools when errors are encountered or the system is not behaving as expected. It explains various warnings and error messages encountered that can be seen in monitor subscriber traces, logs, snmp traps, and/or alarms/thresholds. It may touch on related/relevant areas such as licensing, radius authentication, etc.

Basics of IP pools

Pools must be configured in a destination context in the chassis. This implies that there can be more than one destination context in a chassis depending on what services are provided by the chassis as well as how contexts are configured to handle various groups of subscribers. The destination context is either specified in a subscriber profile, via the command “ip context-name <context name>”, or via the attribute “SN1-VPN-NAME” returned during radius authentication. The destination context is important because that is where the IP pools must be defined. For MIP calls attaching to an FA service, the destination context is not relevant. But for SIP calls (connecting via a PDSN service), or MIP calls (connecting via an HA service), or L2TP calls (connecting via an LNS service), the concept of a destination context is critical, because an IP address is being doled out by the chassis in these scenarios.

Pools can be specified as public, static, and private, and it is important to understand the differences:

- A public pool can provide IP addresses for any kind of call request including ones where a pool name is specified. It will not dole out an address for a static subscriber (see below) unless the qualifier “allow-static-allocation” is specified.
- A static pool can only provide IP addresses for requests where a specific IP address is returned by radius via attribute Framed-IP-Address or is specified in a subscriber template via command “ip address <ip address>” – this latter approach would be more for lab testing and not for production because it would require a separate template for every possible static subscriber which is not practical/feasible.
- A private pool can only provide IP addresses for requests where an IP pool name is specified.

IP pools are often grouped allowing for multiple ranges of addresses to be available for the same set of subscribers. This is done with the “group-name” qualifier when defining the pool. At call setup time pool names are either returned by radius via attribute “SN1-IP-Pool-Name“ or “Framed-Pool” (either works) or are specified in a subscriber template via command “ip address pool name <pool name>”

IP pools reside in contexts that have ip interfaces that lead directly to the internet or to a corporate VPN that itself possibly ultimately leads to the internet. The next hop towards the destination can be optionally specified for subscribers in a given IP pool with the qualifier “next-hop-address”. If not, then the route statements in the (destination) context will be used to route to the next hop address. The other direction coming back from the internet to the context can also be a place where things break related to IP pools. There needs to be properly configured routes in the path between the destination context and the internet that point back to the ip pools or else traffic will not make it all the way back to the context.

IP pools can be temprorarily marked out of service for giving out any new IP addresses, which is called busyout. This can be done during troubleshooting if there are issues with a specific pool and you want to not affect any more users, or when making pool changes where you know in advance that you will be removing a pool and want to minimize the number of subscribers impacted. Busy-ing out the pool will not affect existing subscribers, and due to normal call timeouts, with enough advanced notice, you can busy-out a pool allowing remaining subscribers to eventually timeout (and possibly reconnect to a different pool), resulting in the pool replenishing most of the addresses. The busy-out command is also a config command (versus an exec command), but is a separate config line from ip pool configuration. At the context level in which the pool is defined: "busyout ip pool name <ip pool>".

Viewing status of IP pools

To view information about pools, you must run the commands in the context where the pools are defined, otherwise “No IP Pools are configured for context <cpntext name>” will be returned. The basic version of the command is “show ip pool”. For example, for the following pool definition, with one subscriber connected, run the basic command "show ip pool":

ip pool mippool 20.20.20.0 255.255.255.0 public 0

[destination]CSE02# show ip pool
context destination:
+-----Type:    (P) - Public    (R) - Private    (N) - NAT
|              (S) - Static    (E) - Resource   (O) - One-to-One NAT
|              (M) - Many-to-One NAT
|
|+----State:   (G) - Good      (D) - Pending Delete   (R)-Resizing
||             (I) - Inactive
||
||++--Priority: 0..10 (Highest (0) .. Lowest (10))
||||
||||+-Busyout: (B) - Busyout configured
|||||
|||||
vvvvv Pool Name               Start Address   Mask/End Address Used    Avail  
----- ----------------------- --------------- ---------------  ----------------
PG00  mippool                 20.20.20.0      255.255.255.0    1        253   

Total Pool Count: 1

It is important to be aware of the distinction between the number of subscribers reported (via show subscriber summary or like commands) and the number of used IP addresses reported by this command. For example, in the case of combo chassis that house both FA and HA sessions, there will be two sessions (pdsn-mobile-ip and ha-mobile-ip) for every subscriber that is anchored to both the FA and HA service on the chassis. Another example would be a SIP call (pdsn-simple-ip) extended via a LAC to an LNS (lns-l2tp) that provides the IP address, where the services are all located on the same chassis. Here is an example of a FA/HA combo showing the same IP listed twice:

[destination]CSE02# show subscrber all
+-----Access (M) - pdsn-mobile-ip  (H) - ha-mobile-ip
|     Type:
vvvvvv CALLID   MSID            USERNAME               IP              TIME-IDLE
------ -------- --------------- ---------------------- --------------- ---------
MXCNAM 093d6ae3 0000012345      dave                   20.20.20.4      00h06m33s
H.CNAI 0abae326 -               dave                   20.20.20.4      00h12m29s

Extended information about pools is available and can be useful in troubleshooting.

The verbose qualifier includes among other things, ip pool configuration information that can be overlooked when just starring at a series of (long) ip pool config lines. For the simple example above:

[destination]CSE02# show ip pool verbose
Ungrouped Public Pools:
    Pool: mippool                         20.20.20.0       255.255.255.0  
    Pool Status:       Good
      Type:            Public                  Priority: 0        
      Pool-Tag:        none
      Group:                                       VRF:n/a
      Used:            1                           Free: 253      
      Hold:            0                       Released: 0        
      Addr-Hold-Timer: -                       Limit Exceeded: 0        
      Total Alloc Req: 22                 Total Rel Req: 21       
      Input Label: n/a         Output Label: n/a
      Network Reachability Detection Server: Disabled 
             Unicast Gratuitous-ARP Address: Disabled
                 Nexthop Forwarding Address: Disabled          Vlan ID: n/a
                   Suppress-Switchover-ARPS: Disabled 
                 Send-ICMP-Dest-Unreachable: Disabled 
                   Explicit-route-advertise: Disabled 
          Include-Network-Broadcast-Address: Disabled 
                  Group Available Threshold: Disabled    Clear: Disabled
                        Pool-Free Threshold: Disabled    Clear: Disabled
                        Pool-Used Threshold: Disabled    Clear: Disabled
                     Pool-Release Threshold: Disabled    Clear: Disabled
                        Pool-Hold Threshold: Disabled    Clear: Disabled


Total Explicit Host Routes: 0
Total Pool Count: 1

Here is an example where pools are grouped, and where there is only one pool in this group. Note also the defined group threshold (more on this later).

Group: Group1
    Pool: Pool1                    80.207.128.0     255.255.192.0  
    Pool Status:       Good
      Type:            Private                 Priority: 0        
      Pool-Tag:        none
      Group:           Group1                  VRF:n/a
      Used:            4579                        Free: 11803    
      Hold:            0                       Released: 0        
      Addr-Hold-Timer: -                       Limit Exceeded: 0        
      Total Alloc Req: 4559431            Total Rel Req: 4554852  
      Input Label: n/a         Output Label: n/a
      Network Reachability Detection Server: Disabled 
             Unicast Gratuitous-ARP Address: Disabled
                 Nexthop Forwarding Address: 86.174.216.1      Vlan ID: n/a
                   Suppress-Switchover-ARPS: Disabled 
                 Send-ICMP-Dest-Unreachable: Disabled 
                   Explicit-route-advertise: Disabled 
          Include-Network-Broadcast-Address: Disabled 
                  Group Available Threshold: 40%         Clear: 45%
                        Pool-Free Threshold: Disabled    Clear: Disabled
                        Pool-Used Threshold: Disabled    Clear: Disabled
                     Pool-Release Threshold: Disabled    Clear: Disabled
                        Pool-Hold Threshold: Disabled    Clear: Disabled

    Group Summary:
    Group Used:                         4579
    Group Free:                         11803
    Group Hold:                         0
    Group Released:                     0
    Group Effective Alarm Threshold %: 40%
    Group Effective Clear Threshold %: 45%
    Group Current Usage %:             27.95%
    Group Status:                      Good

There is also a "wide" version of show ip pool that lists additional information such as number of held addresses in the tabular format, which you may find helpful.

vvvvv Pool Name     Start Address   Mask/End Address Used     Hold     Avail    Rel      Free     Group Name
----- ------------- --------------- ---------------  -------- -------- -------- -------- -------- -----------
PG00  mippool       20.20.20.0      255.255.255.0    1        0        253      0        253

Total Pool Count: 1

You can see a list of all possible IP addresses in a pool and their respective statuses with "show ip pool address ...":

[destination]CSE02# show ip pool address pool-name mippool limit 300
+-------- (B) Busyout
|
|+------- (F)-FREE (U)-USED (H)-HOLD (R)-RELEASE 
||
|| Address         NAI/MSID Hash    Hold Timer      Session Start/Disconnect
vv ==============  ================ =============== =========================

Pool: mippool
U 20.20.20.4      f2ed59f3c14e495b         -        Thu Jul 16 08:48:14 2009 

F 20.20.20.1      0000000000000000         -                    -            
F 20.20.20.2      0000000000000000         -                    -            
F 20.20.20.3      0000000000000000         -                    -            
F 20.20.20.5      0000000000000000         -                    -            
F 20.20.20.6      0000000000000000         -                    -            
...        
F 20.20.20.254    0000000000000000         -                    -
  

Troubleshooting

This section covers some possible types of failures related to IP pools as well as possible steps to take.

Problem: Insufficient resources

Mobile IP error code Insufficient Resources
       Message Type: 0x03 (Registration Reply)
               Code: 0x42 (Insufficient Resources)


The disconnect reason code at the end of a monitor subscriber trace and reported by “show session disconnect-reason” will be MIP-proto-error

Note that for Simple IP unlike for MIP, insufficient resources results in PPP negotiation failing with a PPP Term sent to the mobile, which by itself is not very useful because there are many potential causes for this. But, the disconnect reason for such calls will be reported as “No-IPV4-address-for-subscriber” which is useful.

Insufficient resources is often related to IP pools as follows:

- IP pools have actually run out of IP addresses to give out.
     - this would be considered an outage scenario
     - if proper capacity planning and testing has been done, then addresses should not just run out suddenly, and one needs to determine why so many addresses are being allocated. Possible considerations:
            - some other chassis in the network is down or not giving addresses or transport issues in network preventing other chassis from taking calls and overflow is being sent to this chassis
            - configuration changes made to radius or other equipment in network resulting in calls being sent to this chassis
            - new call policy feature enabled on another chassis and pointing calls to this one, or to a Foreign Agent which in turn is connecting to this one for addresses
            - this chassis is experiencing an issue with some services, so that addresses are being requested from this particular pool (group) when they would not normally be or at least not at such a high rate. For example, maybe MIP calls are failing down to SIP abnormally resulting in SIP pools are being exhausted.
     - if an alert-threshold is configured for the pool, then an alarm and SNMP trap should have been triggered to warn of potential pool exhaustion (more on this later)

- IP pool name specified for the subscriber does not exist (applies to both static and non-static users)
     - check that the Radius server is properly configured for the subscriber
     - check that the correct subscriber template is being assigned for the subscriber

- no static IP pool or public pool with policy “allow-static-allocation” exists that can dole out requested IP address
     - check to see if a static ip pool or public pool with policy “allow-static-allocation” exists that contains the ip address requested
     - check that the Radius server is configured with the proper ip address that the subscriber is supposed to be programmed for
     - check to see if the address has already been doled out to that subscriber or a different subscriber
     - show subscriber trace may display the following:
***CONTROL*** 10:35:14:447 Eventid:10083
Sessmgr-17 Failed to allocate static IPv4 address 85.255.191.1 mask 0xffffffff poolname <Static1> for call (errcode=VPN_MSG_STATUS_FAILURE)

- IP pool(s) have been busyout'd (config mode command "busyout ip pool name <ip pool name>") for some reason and so no addresses can be doled out, even though there are many free addresses. One would normally know this but with multiple personnel, locations, and shifts, it's possible that this information has not been communicated to all who should know.

- Note that if the issue has to do with insufficient licenses, the MIP error code will be “0x81 Administratively Prohibited” instead.

Problem: SNMP trap and alarm triggered indicating pools being depleted

- First, this does not necessarily indicate there is a problem, but it is a warning that there could be a problem
- The alert-threshold qualifier for an ip pool allows for the issuing of a warning when the % of ip addresses available in a group of pools or a specific pool have run low. If this alarm is triggering on a regular basis, then first make sure that the setting makes sense – a high (%) value means that it has been programmed to alert even when there are lots of addresses still available (it does NOT mean % used but rather % free, so a lower number points towards exhaustion - this is a common point of confusion)
- If the threshold is set reasonably, then
     - it may simply be that the number of subscribers has increased on a consistent basis that it is time to increase the number of IP pool addresses in the pool(s) and/or to add new pools. If this is done, make sure to obtain a license with increased sessions if necessary.
     - If you know this not to be the case, and the pool usage continues to grow, then there may be a real problem. See earlier discussion on running out of addresses.

- Example SNMP traps, log messages, and alarms:

ip pool Pool1 80.207.128.0 255.255.192.0 private 0 group-name Group1 alert-threshold group-available 40 clear 45

Wed Jun 24 09:20:01 2009 Internal trap notification 238 (ThreshIPPoolAvail) context destination group Group1 threshold 40 measured value 35
Wed Jun 24 13:40:01 2009 Internal trap notification 239 (ThreshClearIPPoolAvail) context destination group Group1 threshold 45 measured value 46

2009-Jun-24+09:20:01.215 [alarmctrl 65201 info] [8/0/550 <evlogd:0> alarmctrl.c:180] [context: destination, contextID: 3]  [software internal system critical-info] Alarm condition: id 50028f41efc10000 (Minor): <76:available-ip-pool-group> has reached or exceeded the configured threshold <40%>, the measured value is <35%>. It is detected at <IP-Pool [vpn/pool: destination Group1 ]>.

2009-Jun-24+13:40:01.044 [alarmctrl 65200 info] [8/0/550 <evlogd:0> alarmctrl.c:273] [context: destination, contextID: 3]  [software internal system critical-info] Alarm cleared: id 50028f41efc10000: <76:available-ip-pool-group> has reached or exceeded the configured threshold <40%>, the measured value is <35%>. It is detected at <IP-Pool [vpn/pool: destination Group1 ]>.

[local]PDSN# show threshold

Threshold operation model: ALARM

Configured thresholds:

        Name:             available-ip-pool-group
        Config Scope:     Context[destination]
        Threshold:        40%
        Clear Threshold:  45%

Active thresholds:

  Name:             available-ip-pool-group
        Config Scope:     Context[destination]
        Threshold:        40%
        Clear Threshold:  45%
        Poll Interval:    300Seconds
        Next Poll Time:   2009-June-24+10:15:00

Outstanding alarms:

       Threshold Name:    available-ip-pool-group
       Alarm Source:      IP-Pool [vpn/pool: destination Group1 ]
       Last Measured:     37%
       Raise Time:        2009-Jun-24+09:20:01

[local]PDSN# show alarm outstanding
Sev Object     Event
--- ---------- ----------------------------------------------------------------
MN  VPN 3      <66:available-ip-pool-group> has reached or exceeded the configured threshold <40%>, the measured value is <35%>. It is detected at <IP-Pool [vpn/pool: destination Group1 ]>.


Problem: Failure to create a ranged pool: “Failure: IP Address Pool too big! (max: 13 bits/255.248.0.0)”

- For example, the following attempt will generate this error: [egress]CSE_2(config-ctx)# ip pool pub003 range 12.0.0.0 12.7.255.255
- Ranged pools by definition include every possible address specified in the range, whereas masked pools automatically exclude the network and broadcast address of the specified mask. So, you may be restricted when specifying a pool using a range depending on the size of the range.

Imported from Starent Networks Knowledgebase Article # 10560

Version history
Revision #:
1 of 1
Last update:
‎01-25-2012 07:38 AM
Updated by:
 
Everyone's tags (1)