Solved: BGP/OSPF and VRF -- Routes missing

jfgobin01 · ‎05-11-2009

Hello guys,

A strange issue I'm facing, which I was able to reproduce in a lab.

3 routers: PE0, CE0 and Internal0.

PE0 and CE0 have 2 vrfs, call them One and Two.

Between PE0 and CE0, we run BGP (not MP-BGP) in 2 different address-family ipv4.

So, in CE0, I have the routes from PE0, learnt through BGP.

CE0 redistribute BGP in OSPF and OSPF in BGP.

Internal0 has no VRF and connects to the 2 VRF of CE0. Internal0 runs OSPF.

On Internal0, I see all routes, the one redistributed from BGP to OSPF being flagged as External2 (ok).

BUT ... on CE0, I can't see the routes from the other VRF if they were learnt from BGP.

Example:

PE0 in vrf One has a route for 1.1.1.0/24 announced through BGP to CE in vrf One.

CE0 in vrf One has the route for 1.1.1.0/24, learnt through BGP.

Internal0 has a route for 1.1.1.0/24, learnt through OSPF as external type 2.

CE0 in vrf Two has NO route for 1.1.1.0/24, however it has an entry in the OSPF database!

An example of what we see (this is the equivalent of the CE0 of the example):

nycertr01#show ip route vrf vrf_secure 10.1.2.1

Routing entry for 10.1.2.0/24

Known via "bgp 65000", distance 20, metric 0

Tag 13979, type external

Redistributing via ospf 30

Advertised by ospf 30 metric-type 1 subnets

Last update from 172.31.0.154 2d02h ago

Routing Descriptor Blocks:

* 172.31.0.154, from 172.31.0.154, 2d02h ago

Route metric is 0, traffic share count is 1

AS Hops 2

Route tag 13979

nycertr01#show ip route vrf vrf_public 10.1.2.1

% Subnet not in table

nycertr01#show ip ospf 10 database external 10.1.2.0

OSPF Router with ID (172.31.0.145) (Process ID 10)

Type-5 AS External Link States

LS age: 1797

Options: (No TOS-capability, DC)

LS Type: AS External Link

Link State ID: 10.1.2.0 (External Network Number )

Advertising Router: 172.31.0.153

LS Seq Number: 8000005B

Checksum: 0x228D

Length: 36

Network Mask: /24

Metric Type: 1 (Comparable directly to link state metric)

TOS: 0

Metric: 1

Forward Address: 0.0.0.0

External Route Tag: 3489725928

nycertr01#show ip ospf 30 database external 10.1.2.0

OSPF Router with ID (172.31.0.153) (Process ID 30)

Type-5 AS External Link States

LS age: 1799

Options: (No TOS-capability, DC)

LS Type: AS External Link

Link State ID: 10.1.2.0 (External Network Number )

Advertising Router: 172.31.0.153

LS Seq Number: 8000005B

Checksum: 0x228D

Length: 36

Network Mask: /24

Metric Type: 1 (Comparable directly to link state metric)

TOS: 0

Metric: 1

Forward Address: 0.0.0.0

External Route Tag: 3489725928

Any idea?

rakesh.hegde · ‎05-18-2009

The router says "Redistributing via bgp 65000" but I still wanted to confirm that under BGP ipv4 address family for vrf_secure you have redistribute ospf 30 vrf vrf_secure match internal external 2 .

-Rakesh

View solution in original post

Giuseppe Larosa · ‎05-11-2009

Hello Jean,

when you see an external LSA that is not installed in the routing table possible reasons are:

a) a route filter applied with distribute-list in inside OSPF process that blocks installation of the prefix in the routing table but it is still in LSDB.

b) the next-hop of the type 5 LSA is indeed another external LSA: OSPF doesn't accept to install an external route with a forwarding address that is another external route

but in your case advertising router is equal to OSPF 30 router-id that comes from the other VRF as could be expected in your setup.

what is the result of

sh ip route vrf vrf_public 172.31.0.153 ?

if it is an external route it is a problem.

Hope to help

Giuseppe

jfgobin01 · ‎05-11-2009

Hello Giu,

It does make sense and I thought of this, so I tried announcing the network between my CE and the ISP's PE in OSPF:

nycertr01>show ip ospf 10 database external 10.1.2.0

OSPF Router with ID (172.31.0.145) (Process ID 10)

Type-5 AS External Link States

LS age: 1192

Options: (No TOS-capability, DC)

LS Type: AS External Link

Link State ID: 10.1.2.0 (External Network Number )

Advertising Router: 172.31.0.153

LS Seq Number: 80000060

Checksum: 0x1892

Length: 36

Network Mask: /24

Metric Type: 1 (Comparable directly to link state metric)

TOS: 0

Metric: 1

Forward Address: 0.0.0.0

External Route Tag: 3489725928

nycertr01>show ip route vrf vrf_public 172.31.0.153

Routing entry for 172.31.0.152/30

Known via "ospf 10", distance 110, metric 4, type intra area

Redistributing via bgp 65000

Advertised by bgp 65000

Last update from 172.30.0.5 on GigabitEthernet0/0.810, 04:27:32 ago

Routing Descriptor Blocks:

* 172.30.0.5, from 172.31.0.153, 04:27:32 ago, via GigabitEthernet0/0.810

Route metric is 4, traffic share count is 1

But no change. Not route filter applied.

The fun part is it seems related to being on the same router: I have another router on the same segment as my CE and the "top" router:

Stroz-Core-4900-A>show ip route vrf vrf_public 10.1.2.1

Routing entry for 10.1.2.0/24

Known via "ospf 10", distance 110, metric 3

Tag Complete, Path Length == 1, AS 65000, , type extern 1

Last update from 172.30.0.5 on Vlan810, 04:13:07 ago

Routing Descriptor Blocks:

* 172.30.0.5, from 172.31.0.153, 04:13:07 ago, via Vlan810

Route metric is 3, traffic share count is 1

Route tag 3489725928

I'm thinking to a bug.

Giuseppe Larosa · ‎05-11-2009

Hello Jean Francois ,

you may need to clear the ospf process or ospf redistribution to trigger a new evaluation of the external route for 10.1.2.0/24.

Also a clear ip route vrf vrf_public * could help.

Hope to help

Giuseppe

jfgobin01 · ‎05-11-2009

Not a bad idea. I'll plan that for an late "single" evening ;-)

I'll keep this post updated.

jfgobin01 · ‎05-15-2009

Hello,

I did the "clear ip ospf nn process" and "clear ip route vrf nn *" this morning, but no change.

At this point, either there is something funny in the router (IE: the router thinking it knows a route for the next-hop bypassing the inter-vrf router and actually not able to use it) or there is a bug.

I think I'll have to open a TAC request.

J.

Giuseppe Larosa · ‎05-15-2009

Hello Jean Francois,

I would try a reload however it looks like a bug.

Hope to help

Giuseppe

jfgobin01 · ‎05-15-2009

Reload would be hard to achieve. I asked our partner to test it and open a TAC if necessary. I'll keep the list posted.

Thanks anyway,

J.

rakesh.hegde · ‎05-16-2009

Hi,

This has got to do with how BGP interacts with OSPF when OSPF is used as the PE-CE protocol in a L3 MPLS environment .OSPF uses "domain-tag" feature for type 5 and type 7 LSAs to prevent route feedback via another PE . If not configured, (domain-tag under vrf ospf process) the router is going to use the BGP AS number to compute the value. The router then uses this value to set the tag for the LSA . VRF "two" on your CE is going to ignore this LSA since it has the same domain tag value (3489725928). This feature is enabled by default when you create vrf aware ospf processes. There are two solutions.

1. Configure different domains tag values under ospf vrf processes on CE using â€œdomain-tag â€œcommand.

2. Since you are running vrf lite Use "capability vrf-lite" under vrf aware process to suppress this check. I think this is much cleaner.

Please rate if this helps.

-Rakesh

Giuseppe Larosa · ‎05-17-2009

Hello Rakesh,

you're right this issue can be solved adding the capability vrf-lite command

as explained in

http://www.cisco.com/en/US/docs/ios/12_0st/12_0st21/feature/guide/ospfvrfl.html

and

http://www.cisco.com/en/US/docs/ios/iproute/command/reference/irp_osp1.html#wp1012376

And also by the domain-tag command

http://www.cisco.com/en/US/docs/ios/iproute/command/reference/irp_osp1.html#wp1013393

Best Regards

Giuseppe

jfgobin01 · ‎05-17-2009

Hello guys,

Rakesh, it works ... partially: now, our CE displays these missing routes. And it even says "redistributing through BGP".

However, when I display the RIB on another site, none of thoses routes are learned.

I guess I have to look at BGP to check how to be sure it announces everything as coming from itself and discards any tag.

Thanks,

J.

rakesh.hegde · ‎05-17-2009

There could be many reasons why you are not able to see this subnet on "anothe site" RIB .

Please do the following.

1. ON CE make sure you are advertising this subnet to PE vrf B

sh ip bgp vpnv4 vrf b nei advertised-routes

2. ON PE verify that

a. this subnet is in the BGP table of vrf b and it's the best route

sh ip bgp vpnv4 vrf b

b. It's in the RIB

sh ip route vrf b

If you see the route on PE VRf b's RIB, vrf b on PE has sucessfully received the route that PE vrf a advertised to CE vrf a, via CE vrf b. This in general would mean that any sites part of vrf b should be able to see theis subnet.

HTH

-Rakesh

jfgobin01 · ‎05-18-2009

Rakesh,

Here is an example of what I see.

Let's take two VRFs: vrf_voice and vrf_secure on our systems:

10.10.2.0/24 is in vrf_voice and learnt from the PE through BGP:

show ip bgp vpnv4 vrf vrf_voice | incl 10.10.2

*> 10.10.2.0/24 172.31.0.150 0 13979 65002 i

It is in the RIB in vrf vrf_voice:

show ip route vrf vrf_voice 10.10.2.1

Routing entry for 10.10.2.0/24

Known via "bgp 65000", distance 20, metric 0

Tag 13979, type external

Redistributing via ospf 20

Advertised by ospf 20 subnets

Last update from 172.31.0.150 3d01h ago

Routing Descriptor Blocks:

* 172.31.0.150, from 172.31.0.150, 3d01h ago

Route metric is 0, traffic share count is 1

AS Hops 2

Route tag 13979

If I look for this entry in vrf_secure:

show ip route vrf vrf_secure 10.10.2.1

Routing entry for 10.10.2.0/24

Known via "ospf 30", distance 110, metric 1

Tag Complete, Path Length == 1, AS 65000, , type extern 2, forward metric 2

Redistributing via bgp 65000

Last update from 172.30.0.21 on GigabitEthernet0/0.830, 21:28:20 ago

Routing Descriptor Blocks:

* 172.30.0.21, from 172.31.0.149, 21:28:20 ago, via GigabitEthernet0/0.830

Route metric is 1, traffic share count is 1

Route tag 3489725928

But it's not advertised:

show ip bgp vnpv4 vrf vrf_secure nei 172.31.0.154 advertised-routes | incl 10.10.2

nycertr01#

Anyway, it's not even in the RIB:

show ip bgp vpnv4 vrf vrf_secure | incl 10.10.2

nycertr01#

So ... BGP is filtering this out. I guess for the same reason. However, there is no "capa vrf-lite" under BGP or BGP/address-family.

J.

rakesh.hegde · ‎05-18-2009

The router says "Redistributing via bgp 65000" but I still wanted to confirm that under BGP ipv4 address family for vrf_secure you have redistribute ospf 30 vrf vrf_secure match internal external 2 .

-Rakesh

jfgobin01 · ‎05-18-2009

router bgp 65000

no synchronization

bgp router-id x.x.x.x

bgp log-neighbor-changes

no auto-summary

!

address-family ipv4 vrf vrf_voice

redistribute ospf 20 vrf vrf_voice

neighbor x.x.x.x remote-as NNN

neighbor x.x.x.x activate

neighbor x.x.x.x default-originate

no synchronization

network x.x.x.x mask 255.255.252.0

exit-address-family

!

address-family ipv4 vrf vrf_secure

redistribute ospf 30 vrf vrf_secure

neighbor x.x.x.x remote-as NN

neighbor x.x.x.x activate

neighbor x.x.x.x default-originate

no synchronization

network 10.150.0.128 mask 255.255.255.128

network 10.150.12.0 mask 255.255.255.0

network 10.150.14.0 mask 255.255.254.0

network 10.150.16.0 mask 255.255.252.0

exit-address-family

!

address-family ipv4 vrf vrf_public

redistribute ospf 10 vrf vrf_public

neighbor x.x.x.x remote-as NN

neighbor x.x.x.x activate

neighbor x.x.x.x default-originate

no synchronization

network 10.150.0.0 mask 255.255.255.128

network 10.150.1.0 mask 255.255.255.0

network 10.150.2.0 mask 255.255.254.0

network 10.150.4.0 mask 255.255.252.0

exit-address-family

By default, does the redistribute catch all or only internal?

J.