VRF issue on 6500 (lengthy post!)

Unanswered Question
Jun 25th, 2008


Below a rather lengthy description of my VRF issue on the 6500 platform. The background is needed to make the situation (and the problem) clear.



I have 2 * 6509 chassis, interconnected with a 10Gbps backbone. In both chassis is an FWSM. The FWSMs are configured in Failover (Active-Active). Some contexts are routed, some are transparent.

The outside interface of a context is connected to both routers (sup720 in 6500). I use HSRP, because the FWSMs do not support OSPF (not good enough, anyway).

For failover, all interfaces on the context is connected to both FWSMs. For this, 1Gbps links are used.

This setup works, but has a big disadvantage.

Assume the Left FWSM is active, and HSRP is active on the Left sup720 (same chassis). Traffic will flow from the customer, to the left FWSM, to the left sup720, and is routed to the destination. So far, so good.

If the destination is reachable through the left sup720, the packet is delivered, and the response will travel the same path in the opposite direction.

If the destination is reachable over the Right sup720, the packet is forwarder over the 10Gbps link, and delivered.

The response will return to the Right sup720, but the OUTside interface of the firewall (the Left FWSM is active!) is directly connected to the Right sup720 too! The packet will use this link (which is 1Gbps) to reach the Left FWSM.

Not only is this asymmetric routing (more difficult to troubleshoot), but there is also a bandwidth issue.

The solution


Unlike static routes, directly connected routes cannot be made 'floating'. The trick is to change the directly connected link into a static route. This can be done with a VRF.

In the lab, I used 2*4506 to test this.

The relevant config would be:


ip vrf FO312

rd 1:312


interface GigabitEthernet1/1

description *** Connected to customer ***

no switchport

ip vrf forwarding FO312

ip address aa.bb.cc.dd

standby version 2

standby 312 ip


interface Loopback312

description *** Anchorpoint for VRF ***

ip vrf select source

ip vrf receive FO312

ip address


ip route vrf FO312

ip route aa.bb.cc.dd Gig1/1 200



router ospf 1

redistribute static



This works as expected on the 4500 platform. In the global routing table, the customer network is a floating static. When the Left router works as expected, the customer network is in OSPF, and will override the static. All traffic to the customer will go through the Left router, and is forwarded to the Left firewall and reaches is destination.

When the Left Firewall or router fails or link to the customer fails, the route disappears from OSPF, and the static will become visible. Traffic is now routed over the Right router and firewall (Firewall switches over through FailOver).

The problem


On the 6500 platform, with Sup720 with IOS 12.2(18)SXF10, the interface-commands:

ip vrf select source

ip vrf receive [VRFname]

are not available.

I can isolate the interface from the global routing table, but I cannot re-establish an IP connection within the same device.

I've tried:

ip route vrf FO312 global

but it is refused because the IP-address is not a next hop (it's this router). Without the 'global' keyword it is accepted, but the target is not in the VRF's routing table, and thus unreachable.

I cannot route to an interface, because VRF routes can only point to point-to-point interfaces or next-hop IP-addresses.

I considered a Tunnel interface, but this solution should be implemented for multiple contexts, and makes the setup quite complicated.

The other router as next-hop is also a bad choice, because this hop will disappear when the other router fails, and this is all about redundancy.

As you can read, I tried different approaches, but I'm a bit stuck here.

Can anyone help?


I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
n.nandrekar Thu, 06/26/2008 - 01:44

Hi Marcel,

I am not sure if I have understood your problem correctly as I have minimal knwledge ofFWSM nd redundancy here. So dont know if the suggestion will even be valid.

Heres a shot...

What if you have a vrf static route with next-hop ip with "global" keyword and another floating route to backup for this one.

To make things clear, 1 route with one router as next hop with admin dist of say 200 and another route to same destination but using the second N.H. with admin dist of 210. If 1 hop disappears due to router failure, the other will pop in.



mvandorp Thu, 06/26/2008 - 03:47

I've tried that, but with the global keyword, the next hop address cannot be on the same router.

It can be a local interface, but only a point-to-point interface.

If the static points to the other router, it will work, but the whole setup is for redundancy. In this case the VRF will be used once the other router fails. It is of no use if the VRF uses an IP-address on the just-failed router as a next-hop.

Thanks for you comment, anyway.


n.nandrekar Thu, 06/26/2008 - 04:43


I didnot mean using the next hop address of the same router but the peer. For this you say, that the vrf would use the next hop of the just-failed-router as next hop and it will not be acceptable.

For this, I am saying taht install the second floating vrf route pointing to the other redundant (now active) next-hop. Once the 1st router fails, the directly connected link will go down making the next-hop not reachable Due to this the static route will be removed from the VRF routing table and the second static route with the different next-hop and higher AD will e installed.

Wont this work?



mvandorp Thu, 06/26/2008 - 05:26

Yes and no.

In normal operation, the Left Router is the active HSRP router. The Left FWSM is the active one, with a default gw to the HSRP address (which is active on the Left Router).

The traffic TO any destination goes as expected, where traffic to destinations behind the Right router will be routed over the 10Gbps backbone.

Return traffic via the Left Router will go as expected, but return traffic over the Right router will be forwarded.

The VRF is only neccessary on the backup path.

When the Left path is active, OSPF will propagate the route to the customer to the Right router, and overrules the floating static.

When the Left router goes down, the OSPF route will disappear, and the VRF-route will become visible. The return traffic will go through the VRF to the customer.

Outgoing traffic need to be routed without the Left router (as this is the just-failed router), so I cannot use the Left router as a next-hop.

I've copied two diagrams for you (see attachment). Only the active Firewall (F) is shown, as the other is not relevant for this issue.

The first diagram is as it works right now, but with the sub-optimal return-path (notice the 1Gbps bandwidth).

The second diagram is as it works on the 4500 platform, where the VRF isolates the outside VLAN from the core router, making floating static routes possible.

The VRF 'knows' about the global router through the 'vrf select source' and 'vrf receive' commands (missing on 6500 platform).

Makes this things clearer?


PS. I really appreciate the time you are willing to spend on this!

n.nandrekar Thu, 06/26/2008 - 05:59

hi Again!

I have never used the vrf select source / recieve commands and didnt know how they worked. But after prelimnary reading, I feel the folowing feature on 6500 could help achieve the similar result.


the feature is "Multi-VRF Selection Using Policy Based Routing (PBR)"

following is an excerpt :

"Enabling Policy-routing Packets for VRF Instances

To enable policy-routing packets for VRF instances, you can use route map commands with the following set commands. They are listed in the order in which the router uses them during the routing of packets.

•set TOS- Sets the Type of Service (TOS) bits in the header of an IP packet.

•set DF-Sets the Don't Fragment (DF) bit in the header of an IP packet.

•set vrf-Routes packets through the specified interface. The interface can belong to the global routing table or to any other VRF instance.

Note The set vrf command is not supported in Cisco IOS Release 12.2(33)SXH1. <<<<<<<<<<<<<<

•set global-Routes packets through the global routing table. This command is useful for routing ingress packets belonging to a specific VRF through the global routing table. "

You could set a policymap on the VRF and set the packets to do a global ip lookup.

Hope this helps (and that I a not wasting your time ...)



mvandorp Thu, 06/26/2008 - 07:09

Hi Niranjan,

No, you are not wasting my time. I'm glad there is someone I can discuss this with...

I've looked into that, too, but didn't get it to work.

I need the following, right?

1) Make an ACL to select interesting traffic. Can be in my case (all traffic entering this interface)

2) Define a route-map 'match'-ing the above ACL and 'set'ting the global router

3) Apply route-map to VRF-interface

Tomorrow-morning I will have an opportunity to test this in the lab.

If you already see something I missed, please let me know; I will add it to the lab-setup.




This Discussion