Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Webcast-Catalyst9k
New Member

Static routes to a SVI's problem

We have some static routes which point to SVI's. We've had a problem recently with the switch going down which the static route is pointing to. The problem is the route is still in the routing table. Is this normal behaviour and why isn't the route removed from the routing table.

16 REPLIES
Hall of Fame Super Silver

Re: Static routes to a SVI's problem

Hello Darren,

the static route is kept in the routing table for the following reasons:

interface SVI is still up/up because there is at least one L2 interface in vlan X in STP forwarding state.

the ARP table still contains an entry for the static router next-hop

you can try to clear the ARP entry to speed up the process.

Hope to help

Giuseppe

Hall of Fame Super Gold

Re: Static routes to a SVI's problem

Darren

Just to add a bit to the response from Giuseppe: yes this is normal behavior. The reason it does this is that the device with the static route still thinks that the static route is valid. After all the outbound interface on that device (through which the static route points) is still working ok. This is a common problem when static routes point out Ethernet interfaces.

If this is a problem for you there is an alternative which you can consider. Cisco introduced a feature to use object tracking to track a static routes and to remove the static route if the next hop is not reachable. This link should provide information to help get you started if you want to do this.

http://www.cisco.com/en/US/docs/ios/12_3/12_3x/12_3xe/feature/guide/dbackupx.html

HTH

Rick

Silver

Re: Static routes to a SVI's problem

Bump !

To me it's not an expected behavior; a static route shouldn't be kept in the routing table if the underlying interface used to reach the next hop is down

here are my traces; I have a static route belonging to a VRF

show run | i ip route vrf

ip route vrf VRF-PF1 4.0.0.0 255.255.255.128 10.101.219.100

This network points to a SVI which is kept in the "shutdown" state

D7400_DC#show ip interface brief | i 10.101

Vlan5                  10.101.219.2    YES NVRAM  administratively down down

Here is the output of the routing table

D7400_DC#show ip route vrf VRF-PF1 static

Routing Table: VRF-PF1

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

       + - replicated route, % - next hop override

Gateway of last resort is 192.168.230.34 to network 0.0.0.0


      4.0.0.0/25 is subnetted, 1 subnets

S        4.0.0.0 [1/0] via 10.101.219.100

D7400_DC#

So the next hop of my static route is bound to a local interface which is down, but the route is still in the routing table (no old entry in the arp cache or anything else)

If the next hop of a static route is bound to an Ethernet interface (instead of a SVI) which is put in a shutdown state; then the route doesn't appear in the routing table.

Any idea about that ?

I'm running 4500 SUP 7E with the latest IOS 3.3.1SG / IOS XE 15.1SG

Hall of Fame Super Silver

Re: Static routes to a SVI's problem

Hello Surya,

the IP routing table of the VRF shows only a static route, no local or connected route is present in confirmation that you have disabled the SVI associated to the VRF.

However, I see a statement that says:

>> Gateway of last resort is 192.168.230.34 to network 0.0.0.0

Have you configured any form of route leaking to this VRF?

If the answer is positive the default route to 192.168.230.34 ( I don't know where it is, it is not a connected route in this VRF as far as we can see)  might have been used to "resolve" the IP next-hop of the static route with destination 4.0.0.0/25.

Hope to help

Giuseppe


Silver

Re: Static routes to a SVI's problem

Hi.

As you could see in the command I issued in the IOS the output is filtered to show only static routes; I do have an OSPF process by which I learn the default but I can ensure it doesn't conflict / resolve the next hop referenced as next-hop with my static route (you could notice it's a dummy route to a dummy public network).

Here was my Arp cache when I took the trace

D7400_DC#show arp vrf VRF-PF1

Protocol  Address          Age (min)  Hardware Addr   Type   Interface

Internet  192.168.230.33          6   0011.0afe.3820  ARPA   GigabitEthernet2/46

Internet  192.168.230.34          1   0001.e794.0601  ARPA   GigabitEthernet2/46

Internet  192.168.230.35          -   2c54.2dd3.0c3f  ARPA   GigabitEthernet2/46

Internet  192.168.230.65          -   2c54.2dd3.0c3f  ARPA   Port-channel2

Internet  192.168.230.66         50   2c54.2dd3.0aff  ARPA   Port-channel2

D7400_DC#

As I said it's clear now that if a static route points to an SVI which is down; the route still appears in the routing table; if it points to a physical Ethernet interface; the route is not shown in the routing table.

I'll try this again tomorrow on my 4500 / IOS XE 15.1 and then on a 3560 running 12.2(55); maybe the behavior changes according to the IOS version.

It's clearly a big problem as I have 2 chassis; I redistribute all my static routes into my OSPF processes with different costs (to get an active/standby design; I redistribute all static routes with a metric of 50 / E2 on the first chassis and a metric of 100 / E2 on the second unit); so as I have to put into a "shutdown" state all my SVI to prepare my migration; the chassis are still anouncing the static routes (which are being announced in OSPF; as they're present in the routing table; even if all next-hops are unreachable as all SVI are down).

I've never noticed this before.      

Silver

Re: Static routes to a SVI's problem

Hi.

Finally you're probably right; I'll have to investigate furthermore but the problem disappears when I disable OSPF.

So the status of a configured static route in the "live" routing table is not only related to the status of the underlying interface (but it should be);

The correct algorithm for the IOS would check if the nexthop of a configured static route is reachable through a "directly connected" network instead a remote network. Can someone from Cisco confirm ?

Hall of Fame Super Silver

Static routes to a SVI's problem

Hello Surya,

the issue arises from the fact that  Cisco IOS allows to configure static routes with not directly connected IP next-hop invoking routing recursion.

In your scenario when you shut the SVI, the static route next-hop is searched in the VRF routing table and the default route provided by OSPF can be enough to make the route still alive.

At this point, you should try to use reliable static routing with object tracking, so that the static route is removed when the IP SLA with destination = IP next-hop fails.

see

http://www.cisco.com/en/US/docs/ios/12_3/12_3x/12_3xe/feature/guide/dbackupx.html

Hope to help

Giuseppe

Silver

Static routes to a SVI's problem

So for each configured static route; the best practice is to track the status of the underlying interface owning the next-hop ? it sucks !

Can you explain this point in detail ? it sounds to me as a nonsense Except for static route leaking between VRF ?

Cisco IOS allows to configure static routes with not directly connected IP next-hop invoking routing recursion.
Cisco Employee

Re: Static routes to a SVI's problem

Hello Giuseppe,

In your scenario when you shut the SVI, the static route next-hop is  searched in the VRF routing table and the default route provided by OSPF  can be enough to make the route still alive.

Hmmm... A next hop of a static route should never be resolved using a default route. It can be resolved through any more specific route (i.e. a prefix length of at least /1) but Cisco routers should never use a default route to resolve some other route's next hop. That was my understanding and experience. Are you suggesting that the behavior is different?

Thank you!

Best regards,

Peter

Silver

Re: Static routes to a SVI's problem

Hi Peter.

What you say makes sense; the subnet which owns the nexthop address is announced by several routers in my OSPF backbone.

The network owning the ip address of my next hop within my static route is learnt through OSPF.

Hall of Fame Super Silver

Static routes to a SVI's problem

Hello Peter,

you are right I have made a wrong assumption when I had not enough details for the case we were discussing

Rated as it deserves  (I try to pay my debts ...)

As Surya has added in his last  post the IP subnet of the next-hop is learned in OSPF in the VRF so when the local node SVI is disabled the IP next-hop is resolved with the specific OSPF route and the IP next-hop becomes not directly connected.

Surya:

the recursion is actually this: the IP next-hop of the static route resolved by another route

Best Regards

Giuseppe

Silver

Static routes to a SVI's problem

Ok thank you all I understand now.

Is there any real application of route recursion if the next hop is not directly connect ?

In my case it creates a blackhole.

Cisco Employee

Static routes to a SVI's problem

Hi Surya,

Is there any real application of route recursion if the next hop is not directly connect ?

Sure - think of BGP, for example. In iBGP, the next hop is the ASBR either in the next AS or in your own AS (depending on whether you're using the next-hop-self). This concept has many applications also in MPLS world, like MPLS L3 VPNs, 6PE, 6VPE, etc.

Best regards,

Peter

Cisco Employee

Static routes to a SVI's problem

Hello Giuseppe,

No need to apologize, I wasn't pointing out any inaccuracy - I just got intrigued by your statement. As it happens so often, things taken for granted are often quite different.

And most certainly, there is no debt on your part! On the contrary, I feel indebted to you for many, many things. Are you perhaps coming to Live! 2013 to London? It would be awesome to meet you there...

Best regards,

Peter

Silver

Re: Static routes to a SVI's problem

Hi all.

I could make some tests on a 3560 12.2(55)SE6 ip services :

  1. When a network owning the nexthop ip address is not announced through OSPF it's ok; static routes don't appear in my routing table.
  2. As soon as I announce a route to the nexthop network through OSPF; my static routes appear in the routing table.
  3. At some point I withdraw the subnet from the OSPF network; so the subnet of the nexthop is not propagated and OSPF converges well but...
  4. it takes near 1 minute for the IOS to withdraw all the static routes from the routing table; have you ever seen that kind of convergence times ?

Here is the output of a "debug ip routing"

when I add a remote network in the OSPF backbone; static routes are installed in the routing table; the next hop is 10.101.219.103 and for my test I add somewhere in the OSPF backbone a route to 10.0.0.0/8

*Mar  1 04:30:21.847: RT(VRF-PF1): add 10.0.0.0/8 via 192.168.230.36, ospf metric [110/11]

*Mar  1 04:30:27.820: RT(VRF-PF1): add 0.0.0.0/0 via 10.101.219.103, static metric [1/0]

*Mar  1 04:30:27.820: RT(VRF-PF1): default path is now 0.0.0.0 via 10.101.219.103

*Mar  1 04:30:27.820: RT(VRF-PF1): new default network 0.0.0.0

*Mar  1 04:30:27.820: RT(VRF-PF1): add 55.8.56.162/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:30:27.820: RT(VRF-PF1): add 66.237.42.30/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:30:27.820: RT(VRF-PF1): add 86.65.172.65/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:30:27.820: RT(VRF-PF1): add 134.26.27.96/27 via 10.101.219.100, static metric [1/0]

*Mar  1 04:30:27.820: RT(VRF-PF1): add 147.204.2.0/24 via 10.101.219.100, static metric [1/0]

*Mar  1 04:30:27.820: RT(VRF-PF1): add 155.94.56.0/22 via 10.101.219.100, static metric [1/0]

*Mar  1 04:30:27.820: RT(VRF-PF1): add 172.16.21.50/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:30:27.820: RT(VRF-PF1): add 172.16.21.51/32 via 10.101.219.100, static metric [1/0]

[output suppressed]

At some point I withdraw the 10.0.0.0/8 from the OSPF network; look to the timestamps !!

*Mar  1 04:34:33.724: RT(VRF-PF1): del 10.0.0.0 via 192.168.230.36, ospf metric [110/11]

*Mar  1 04:34:33.724: RT(VRF-PF1): delete network route to 10.0.0.0

*Mar  1 04:35:27.855: RT(VRF-PF1): del 0.0.0.0 via 10.101.219.103, static metric [1/0]

*Mar  1 04:35:27.855: RT(VRF-PF1): delete network route to 0.0.0.0

*Mar  1 04:35:27.855: RT(VRF-PF1): del 55.8.56.162/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:35:27.855: RT(VRF-PF1): delete subnet route to 55.8.56.162/32

*Mar  1 04:35:27.855: RT(VRF-PF1): delete network route to 55.0.0.0

*Mar  1 04:35:27.855: RT(VRF-PF1): del 66.237.42.30/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:35:27.855: RT(VRF-PF1): delete subnet route to 66.237.42.30/32

*Mar  1 04:35:27.855: RT(VRF-PF1): delete network route to 66.0.0.0

*Mar  1 04:35:27.855: RT(VRF-PF1): del 86.65.172.65/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:35:27.855: RT(VRF-PF1): delete subnet route to 86.65.172.65/32

*Mar  1 04:35:27.855: RT(VRF-PF1): delete network route to 86.0.0.0

*Mar  1 04:35:27.855: RT(VRF-PF1): del 134.26.27.96/27 via 10.101.219.100, static metric [1/0]

*Mar  1 04:35:27.855: RT(VRF-PF1): delete subnet route to 134.26.27.96/27

*Mar  1 04:35:27.855: RT(VRF-PF1): delete network route to 134.26.0.0

*Mar  1 04:35:27.855: RT(VRF-PF1): del 147.204.2.0/24 via 10.101.219.100, static metric [1/0]

*Mar  1 04:35:27.855: RT(VRF-PF1): delete subnet route to 147.204.2.0/24

*Mar  1 04:35:27.855: RT(VRF-PF1): delete network route to 147.204.0.0

Should I open a case ? I don't have a 4500 sup7 or any switch with IOS 15 to test currently; right now my 4500 are connected on a production network so I can't play with it anymore.

Also I did another test : when I add a valid static route to 10.0.0.0/8 it takes more than 30 seconds to ba taken in account by the IOS for making all other static routes up in the routing table :

*Mar  1 04:45:52.807: RT(VRF-PF1): add 10.0.0.0/8 via 192.168.230.36, static metric [1/0]

*Mar  1 04:46:27.955: RT(VRF-PF1): add 0.0.0.0/0 via 10.101.219.103, static metric [1/0]

*Mar  1 04:46:27.955: RT(VRF-PF1): default path is now 0.0.0.0 via 10.101.219.103

*Mar  1 04:46:27.955: RT(VRF-PF1): new default network 0.0.0.0

*Mar  1 04:46:27.955: RT(VRF-PF1): add 55.8.56.162/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:46:27.955: RT(VRF-PF1): add 66.237.42.30/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:46:27.955: RT(VRF-PF1): add 86.65.172.65/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:46:27.955: RT(VRF-PF1): add 134.26.27.96/27 via 10.101.219.100, static metric [1/0]

*Mar  1 04:46:27.955: RT(VRF-PF1): add 147.204.2.0/24 via 10.101.219.100, static metric [1/0]

*Mar  1 04:46:27.955: RT(VRF-PF1): add 155.94.56.0/22 via 10.101.219.100, static metric [1/0]

*Mar  1 04:46:27.955: RT(VRF-PF1): add 172.16.21.50/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:46:27.955: RT(VRF-PF1): add 172.16.21.51/32 via 10.101.219.100, static metric [1/0]

then I withdraw the static route to 10.0.0.0/8 it takes more than 30 seconds to flush the static routes which are blackholed

*Mar  1 04:48:56.710: RT(VRF-PF1): del 10.0.0.0 via 192.168.230.36, static metric [1/0]

*Mar  1 04:48:56.710: RT(VRF-PF1): delete network route to 10.0.0.0

*Mar  1 04:49:28.083: RT(VRF-PF1): del 0.0.0.0 via 10.101.219.103, static metric [1/0]

*Mar  1 04:49:28.083: RT(VRF-PF1): delete network route to 0.0.0.0

*Mar  1 04:49:28.083: RT(VRF-PF1): del 55.8.56.162/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:49:28.083: RT(VRF-PF1): delete subnet route to 55.8.56.162/32

*Mar  1 04:49:28.083: RT(VRF-PF1): delete network route to 55.0.0.0

*Mar  1 04:49:28.083: RT(VRF-PF1): del 66.237.42.30/32 via 10.101.219.100, static metric [1/0]

*Mar  1 04:49:28.083: RT(VRF-PF1): delete subnet route to 66.237.42.30/32

*Mar  1 04:49:28.083: RT(VRF-PF1): delete network route to 66.0.0.0

The delay to withdraw the static routes is extremely long; so what I learn from these tests is that tracking the status of underlying interfaces for static routing is mandatory; and it's not explained in any course !

Cisco Employee

Re: Static routes to a SVI's problem

Hello Surya,

The static routes are being revalidated for installation or purge from the routing table using a periodic IOS process. By default, this process runs every 60 seconds which should explain the delay you have seen. It is possible to shorten this interval using the global configuration command

ip route static adjust-time seconds

with the seconds argument in the range between 1 and 60. You may want to shorten this interval but if you have a large count of static routes, shortening this interval may lead to higher CPU utilization. Read more here:

http://www.cisco.com/en/US/docs/ios-xml/ios/iproute_pi/command/iri-cr-a1.html#GUID-9EC9E26A-0012-4332-A1D2-0710C551D798

Best regards,

Peter

2499
Views
10
Helpful
16
Replies
CreatePlease to create content