We have the OSPF network shown in the drawing and I am trying to understand what OSPF should be doing.
When the HQ internet is lost, the higher AD of the MPLS edge router flips the gateway to that side for DR internet. That works.
The 525 and 515 PIX firewalls should be getting the default route from the edge router and distributing this to the inside Network.
The 525 firewall is working and distributing the default to the inside network.
The 515 sees the default advertised from the edge router, but uses the default it is getting from the 6509 switch (which is getting it's default from the 525 PIX, gotten from the edge router).
I am trying to understand what OPSF is doing with the two processes in the PIXs and why the 515 prefers the inside to the outside.
I realize the PIX may be confused with the 525 default information originate, which is telling everything on the inside to use it as the default route, while the edge router is also telling the 515 it has the default route.
So, the 6509 is advertising the 525 to everyone internally as the default correct?
But why wouldn't it just as equally use the edge router if it is advertising as a gateway?
Is there a guide to understanding the ospf database and how a device prefers a particular route with that database?
Check the Router-ID from the 6509 and the internet router.
From your GIF file, the internet router's RID is 126.96.36.199. If the 6509's RID is higher, that route will be the preferred route.
You design is a bit odd with the introduction of multiple OSPF processes. Can you elaborate on that design ?
Thanks for the reply.
Internet router ID is higher than 6509 router ID
A CCIE designed this and is not availabe to ask questions.
Apparently during the design, there was some difficulty in getting the adjacentcy to form on the PIX firewalls to the internet router.
TAC suggested the second process, it was implemented and the notes say the desired goal was achieved.
Problems still exist in the design, because there is a static default route to the internet router for Internet connectivity in the 515 PIX.
When that is removed, the PIX routes to the inside for the gateway route.
So the RID determines the prefered route when more than one path is taken?
Can you explain, or point me to a better understanding?
Why would the 525 PIX work as expected by prefering the internet router?
Are there any documents that show how to decipher a database, or why a device is choosing a particular route?
I duplicated your environment and I don't see a problem with the OSPF default route.
Keep in mind, I use IOS routers instead of PIXes.
I can't find any document on understanding the OSPF database. The best book in the subject is "Routing TCP/IP Volume I by Jeff Doyle".
In the meantime, let's verify the 515 has actually a neighbor relationship with the internet router.
show ip os nei
show ip os data | i 0.0.0.0
from the 4 devices in question will help .
I have verified the neighbor relationship with the internet router and both pixes.
The internet router shows:
pix525 as full/dr
pix515 as full/bdr
Pix 515 shows:
internet router as full/drother
pix525 as full/dr
6509 as full/dr
Pix 515 shows:
Link ID ADV Router Age Seq# Checksum Tag
0.0.0.0 188.8.131.52 (internet router) 63 0x800000ae 0x18a4 1
What about default-information-originate?
How would that be affecting the process here?
Pix 515 shows:
Link ID ADV Router Age Seq# Checksum Tag
0.0.0.0 184.108.40.206 (internet router) 63 0x800000ae 0x18a4 1
This output is contradicting what you stated in the initial post. The PIX 515 sees the default route from the internet router via OSPF. It's not showing another 0.0.0.0 from the 6509 in the OSPF database so perhaps you have a static or dynamic route injecting the 0.0.0.0 into the 515 ?
The default-information originate will advertise a default route to its neighbors.
Currently, you have a default-information originate from the internet router back to the PIXes (good design), then you created multiple OSPF processes in the PIXes (not such a good design). On the second process, you have a default-information originate in the 525 towards the 6509. The 6509 only knows to get out to the internet via the 525, what's the use for the 515 ? No internet traffic is leaving thru there.
You are also advertising the DMZ subnet into the internal network (OSPF 2) but the DMZ subnet does not know how to get to the internal network.
I feel like I'm missing something without seeing the actual outputs that I requested.
The PIX is showing the internet router as source of the default route, oh, I see as I am writing this.
The PIX 515 does indeed have a static route to the internet router. I guess that is why it is in the database?, if I remove it, it will point to the 6509 for the default route.
I will try and get the actual outputs while the static route to edge router is removed.
The 515 is actually holding a seperate server for customer Internet transactions. This server belongs to a different department.
Also internal users get to the server off of the 515.
Additionally, when there is an Internet circuit failure in the HQ site, outside customers will get to the server from the DR Internet. This traffic will travel across the inside network the the DMZ on the 515.
The 525 is holding another web server for customer access. I do not know why they were seperated.
So, when a route is a static route on the PIX515, it will show up in the OSPF database on that PIX515?
Does the default-information-originate advertise the default route to everyone in the OSPF process or just the neighbors?
Why did you say the DMZ does not know how to get to internal network?
Something else I wondered, the internal routes are showing up on the edge router, which I think is not such a good idea, should they be filtered from the edge router? If the PIX outside interfaces were passive, wouldn't they still receive the default route from the router, just not advertise the internal routes to it?
I edited the real ip addresses and was reluctant to post them here, the reason for not showing the real outputs, is there some other way for me to get them to you to take a look?
A static default route won't be shown in the OSPF database, this route is learned from the default-information originate from the internet router.
Checking your config once again, I see how the DMZ Server is able to connect to internal users. You have the DMZ interface from the PIX on both OSPF domains, ouch this is too messy.
The default-information originate will advertise the default route to every single OSPF router running in the same OSPF domain. Since you have two domains, it's more complex.
If you change the outside interface on the PIX to passive, they won't become neighbor.
Ideally, you should run one OSPF domain between the internet router, the 2 PIXes and the 6509 switch. You can throw in your MPLS router as well.
Only one default-information originate should be issue and it should come from the internet router. The MPLS router can remain with its default-information originate (weighted higher than the current one).
I appreciate the replys.
So that brings up the question (since the default from the internet router is in the pix515 database), why does the 515 use the 6509 as a preferred path to the internet router?
Or or maybe how is it possible?
On the pix DMZs, they are seperate DMZs, I may have pasted one to the other, but that is all working ok.
I will like to see the OSPF database when that takes place. Currently, there is only one default route in the OSPF database and it's pointing to the internet router.
The config you posted may have missing information, and that missing information may be the cause of the problem. Sanitize your config by changing the IPs but keep the logic in the config, post it again...
Are you talking about missing information in the ospf config?
I can email them to you, or post the entire config here and just delete it as soon as you have had a chance to look at it.
It is perplexing why the 515 is not using the edge router as the default gateway and why the 6509 is not showing up in the database.
I have noticed on the both PIXs, the the 525 (working) shows only one process as active.
The non working is showing both processes as active.
I am wondering if the working PIX actually only has one process that is working ok (both processes have the outside subnet configured in them).
The non working PIX has two active processes, but one process is not distributing the outside subnet into the second process.
This pix does not have the ouside subnet configured in the inside process closest to the 6509 switch.
I have attached this showing the real ip addreses
Yes, I'm talking about missing information in the OSPF config.
Please post the sanitized config so it can be of value for other members of the forum.
It is perplexing what we are seeing here and if it's affecting your network, I suggest you get someone onsite or open a TAC case.
They will be able to get the whole picture, Layer1 thru Layer7.
Based on your latest attached file, it seems the 525 shouldn't be working at all !
On the sanitized config, they are pretty extensive, what should I remove?
On the ospf components, I gave you everything.
Usually this forum is more valuable to me than TAC.
I opened two cases with TAC.
I didn't get anywhere with the PIX engineers.
Basically one told me to duplicate the config from the working pix to the non working pix including the "default-information-originate"
The other one told me to upgrade the 525 image becase of a bug with the failover components.
posted is the ospf from both PIXs
I was thinking the 515 did not have the edge subnet in both processes, but i guess it does.
No glaring problems in your last attached file.
Let's see a complete output from these commands on both PIXes:
show ip os nei
show ip os data
show ip os
Ok, here is the deal.
1) Your OSPF database is in bad shape. It can be a combination of using multiple OSPF processes in addition to a bug within the PIX.
2) The 525 OSPF database only shows LSAs in OSPF Process 2. OSPF Process 1, and this is the important one, only shows one LSA - itself.
3) To add insult to injury, 525 is the DR for that segment so any OSPF communication between the 515 and the internet router needs to go via this PIX. It's strange the 515 and Internet Router OSPF Process 1 database contains the correct LSAs.
Here is my suggestion, move the DR over to the internet router. You can do this by changing the Router-ID in the internet router to a value greater than the 2 PIXes. Use Router-ID 255.255.255.255 for instance.
If you can afford the downtime, reboot the 525.
All this while, you were thinking the problem was the 515, but it was the 525.
This setup is ideal for a CCIE Lab but never for a production environment :)
wow thanks a ton for this information.
sorry for this but
what tells you that:
The ospf database is in bad shape?
Why is process 1 on the 525 the "important one"?
What tells you the 515 and internet router contain the correct LSA's?
How do you know they are correct?
Why do you think this is strange?
The 525 has a failover pair and if I reboot it, it will failover to the standby. Is this what you are talking about, or reboot both of them?
Just compare the OSPF database on the 515 and 525, they are different. Also, compare the OSPF Process 1 on the 515 and the internet router, they are picture perfect.
The 525 does not have the same database on OSPF Process 1.
I consider this Process important because you are bringing the external and default route via this process. I highly suggest you consider consolidating your OSPF database into one process. This is too complex without any need.
I suggest reboot both 525s and make sure to move the DR over to the internet router.
However, I still feel the only way to fix this for good is by consolidating OSPF into one process.
I really appreicate the time you have spent in helping me understand this.
On the ospf provess change, is all that is needed is to remove on of the processes and make sure all needed interfaces are in the single process?
I guess I do not know what you mean by picture perfect.
Are you saying that because they are the same on the 515 and router?
And the 525 is showing one process with no activity?
Also, by rebooting the 525s, this will cause the DR to move to the newly reconfigured RID on the internet router and nothing else needs to be done?
Picture perfect means both databases are identical and that's good.
The 525 is only showing LSAs in OSPF Process 2. I believe it has do to with the fact that you have the external interface enabled on both processes. This may be creating the routing problem and will be remedied if you go with one OSPF process.
What's strange is that you have a similar configuration in the 515, yet is behaving as it should. It can be a bug in the 525, not sure since I don't deal with PIX much.
How do you feel about changing this configuration ?
As for the rebooting question, once the DR goes down, a new election takes place and the router with the highest RID will become the DR. Make sure to have the highest RID in the internet router before rebooting the 525.
I am all for changing it since it is a problem and it could contribute to other problems down the road.
I believe the setup in this company has been overly complicated for some reason.
There have been other issues that have been corrected already.
For example, when I first got here about four months ago, the dynamic failover did not work at all, one of the issues was that all devices that had OSPF configured (including the 6509 siwtches) had the "default information originate" in the ospf config.
I do not think the CCIE did that, but the guy working here before I came did it. There were also static default routes in 6509s to the 525 pix.
I have some of the design notes and the CCIE engineer said he could not get the adjacentcys to form with one process, called TAC and they suggested it, maybe the pix TAC guys do not have too much experience with ospf.
But I want it to be designed correctly, one reason for my persistant seemingly never ending and annoying questions.
I do not have the expertise you guys have to figure all of this out on my own, and I want to understand why things are doing what they do.
I have a PIX 515 I can test some of this on.
I really appreciate all of the help and your patience. I hope you know how valuable I think this forum is and how much I have learned from it.
I got started late in life, so just about the time I get so that I know something, it will be time to retire.
One more thing, should the inside routes be filtered from the ouside network ?
If so, what would be the best way to do it?
The DMZ network, as we referred the outside network, needs to be able to reach the inside network so the routes are needed in that space.
The internet router also have the routes but they aren't advertised to the internet since you aren't redistributing OSPF into BGP.
Ideally, you would setup a NAT on the PIX so routes from the inside network aren't advertised to the DMZ.
Are you saying the LSAs should all match in all devices running ospf?
For example ,I should see the same number of LSAs and checksum should match on all of them?
And how did you know the process 2 on the 525 showed the LSA as itself?
Start time: 00:00:57.329, Time elapsed: 2w0d
Supports only single TOS(TOS0) routes
Supports opaque LSA
Supports Link-local Signaling (LLS)
Supports area transit capability
It is an autonomous system boundary router
Redistributing External Routes from,
Router is not originating router-LSAs with maximum metric
Initial SPF schedule delay 5000 msecs
Minimum hold time between two consecutive SPFs 10000 msecs
Maximum wait time between two consecutive SPFs 10000 msecs
Minimum LSA interval 5 secs
Minimum LSA arrival 1000 msecs
LSA group pacing timer 240 secs
Interface flood pacing timer 33 msecs
Retransmission pacing timer 66 msecs
Number of external LSA 74. Checksum Sum 0x2932C7
Number of opaque AS LSA 0. Checksum Sum 0x000000
Number of DCbitless external and opaque AS LSA 0
Number of DoNotAge external and opaque AS LSA 0
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Number of areas transit capable is 0
External flood list length 0
IETF NSF helper support enabled
Cisco NSF helper support enabled
Number of interfaces in this area is 7 (5 loopback)
Area has no authentication
SPF algorithm last executed 1w4d ago
SPF algorithm executed 7 times
Area ranges are
Number of LSA 18. Checksum Sum 0x0B1038
Number of opaque link LSA 0. Checksum Sum 0x000000
Number of DCbitless LSA 0
Number of indication LSA 0
Number of DoNotAge LSA 0
Flood list length 0
OSPF speaking routers participating in the same area must have an identical view of the network. That's what makes OSPF a link state protocol, it keeps track of all links via its database.
If you look in the 'show ip os data' from the 525 under OSPF Process 2, it only shows one LSA. That LSA is type 1 (router link) and the IP is the 525's (itself).
Edit: Correction, the 525 Process ID 1 is the one showing only one LSA. Process ID 2 is working as it should.
One last thing on this:
The RID can be any ip address, it doesn't have to be an active interface on the router?
And who do you want the DR to be in the OSPF process?
It can be any IP address and it doesn't have to be an IP address on the router. If you don't manually enter the RID, the router will select the highest loopback address. If the loopback address is missing, it will select the highest ip address from any of the physical interfaces.
Ideally, the internet router should be the DR.
wow Edison, don't you ever rest?
One last question and I will leave you alone:
Does the lsa hold the ospf database?
if so, is that all it does?
I'm resting now, you should see when I'm really working :)
LSAs are link-state advertisements that describe the OSPF topology. LSAs are stored in the OSPF database and there are several types:
LSA Type 1 is the Router LSA. This LSA informs other OSPF speaking routers about its existence
LSA Type 2 is the Network LSA. This LSA informs other OSPF speaking routers about the existence of a DR (Designated Router).
LSA Type 3 is the Summary LSA. This LSA comes from an ABR and it informs other OSPF speaking routers about links located in other areas.
LSA Type 4 is the ASBR Summary LSA. This LSA informs other OSPF speaking routers about the existence of an ASBR. It's needed for reachability to external routes.
LSA Type 5 is the External LSA. This LSA informs other OSPF speaking routers about routes being redistributed into OSPF.
Last but not least, LSA Type 7. This is the NSSA LSA and you only get to see this LSA when you configure not-so-stubby areas.
There are other LSAs but not supported by the Cisco IOS.
Based on the OSPF database you posted, your network only has LSA Type 1 (Router Links), Type 2 (You have DR/BDR) and Type 5 (You have a redistribution into OSPF from BGP and other OSPF Process ID).