Core Network Design

[anis_cisco]

Hello,

We are in stage of designing the core netwokring for one of our customer.

Our customer is a Finance Institute, having two building. Each building has two 6509 core switches. Total we have 4 core switches. Both buildings are connected to each other through Fiber Cable.

We have planned to configure VSS between two core switches of each building & run OSPF area 0 between building to avoid loop.

We have to design Vlan's based on departments, labs & classrooms.

Our issue is, The Institute has single Admin, Management Department for both buildings, how we should design our vlan (for Admin & Managment depart) so that it should be reachable for both buildings & should not have conflict.

Can any one share best practice for vlan design in such type of design, what naming convention we should use ?

Regards,

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Why do you really, really need to extend the same VLAN across the two buildings?  In fact, with routed switches, ideally you often don't want to extend a VLAN beyond any one L2 switch.

6500's generally will forward traffic between hosts on different subnets as quickly as they would between hosts on the same subnet.

[Vasilii Mikhailovskii]

Hello.

I agree with Joseph - there is no reason to extend VLAN over the switches. The [most] best practice is not to have L2 interface between devices (but client interfaces :)).

Do you use any low-end switches that do not support routing? This could be only case for L2 extension.

I think you are using "Admin/management" VLAN for security reasons... I guess for configuration simplicity you could assign some /20 summary for all the admin vlans and assign /24 per building or floor.

Even though if you won't extend your L2, I would recommend not to reuse VLAN IDs between switches. I think names are really not important, but VLAN IDs you will be using.

Our enterprise has a document that states what ID may have any VLAN by its' roles.

[Jeff Van Houten]

How many fibers between the buildings? Do you have the capacity to configure both a routed and trucked port between both buildings? Not that that would be a final recommendation of mine, but....

And, are the fiber paths between the buildings redundant? If not, then a cut impacts both routed and L2 configurations equally ( most likely). So if the paths are non-redundant, the added complexity of an L3 design will not protect you from a back-hoe and may not be worth it. Short of a misconfiguration, how likely is an stp re-convergence?

And why Ospf? If it's all Cisco eigrp is recommended. Convergence is much quicker.

And why route anyway? You already said you were going to subnet based on department and room. Do you have a switch per room? If so, I agree with Joseph. If not, I think there is still room for larger L2 configurations.

Sent from Cisco Technical Support iPad App

[anis_cisco]

Thanks for comments,

Joseph,

This is something requested by customer that they have centralized Admin & Management Department for both Campus (i mean both building) & they want to have same vlan.

Mikhailov,

They have 3750 POE as edge switches & we have planned it to use in access layer. 6509 would be core switch for both building, which will perform intervlan routing between there vlans.

Jeff,

Customer has redundant cross connectivity of fiber between buildings.

We dont want to configure Trunk between both buildings,if we will configure trunk then we have to be carefull for Spanning tree calculation for root switch issue. To avoid looping & root switch election we want to go for routing protocol either OSPF or EIGRP. We wont have switch in each room, we have three switch in each floor (in one cabinet at each floor) & all users are terminated on it.

I was looking for solution & i found a feature call QinQ ( spread Vlan over WAN). Can any body advise that does this feature fit for my requirement ?

Regards,

[Joseph W. Doherty · Accepted solution]

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

Joseph,
This is something requested by customer that they have centralized Admin & Management Department for both Campus (i mean both building) & they want to have same vlan.

Well, they should be talked out of it.  But if you can't, just trunk between two buildings and extend the VLAN(s).

[daniel.dib · Accepted solution]

Q-in-Q is a feature for adding an extra VLAN tag so that you can carry multiple VLANs with one outer VLAN tag and forward based on that. It is kind of a tunneling technique.

If they want to spread VLANs then you will have a large L2 topology, that is generally not recommended but sometimes a must. Depending on platform you could carry VLANs with EoMPLS so that you at least terminate the L2. That would require a MPLS core though.

Daniel Dib
CCIE #37149

[Marvin Rhoads · Accepted solution]

I agree with Joseph - explain that having the same VLAN across multiple buildings compromises the design and is most definitely not best practices. Q-in-Q is not designed to solve this sort of problem. MPLS makes the design way more complex than necessary.

This is actually a very simple setup:

Three switches per floor in a stack. Fiber uplinks with ports configured to be members of a single port channel and trunking from stack master and member #2 or 3 to each of the 6509s in the VSS core for that building. 6509 VSS pair (multi-chassis etherchannel) set to be spanning tree root for each building. Routing between VLANs within the building done via layer 3 interfaces (SVIs) on the VSS cores. Second building setup the same. Routed ports (one on on each member of a given building's 6509 VSS core) connecting to the other building and performing equal cost multipath.

You didn't mention remote sites and Internet access but those would need to be taken into account as well.

[anis_cisco]

Hello gents,

Thank you so much for all of your feedbacks & suggestion.

After having long discussion with customer I am back with final design.

Please first have alook on attachment.

As joseph suggested, customer is agreed on to configure separate subnet for each building & then routing between them.

Here I have 2 main concern that i need to follow or im worrying about:

1 - I need to design a standard protocol or Layer 3 routing to enable communication between 4 buildings, right now they are upgrading network for 4 buildings & later on they will upgrade network for 16 buildings more. So we have to use an standard that should be same between all buildings.

2 -  It should be Layer 2 loop free design.

As in attachment it shows that OSPF will configure between Building 1 & Building 2. Purpose to enable OSPF is to keep the Building 1 & Building 2 as layer 2 loop free. What else options do we have as Layer 3 communication between buildings ? Keep this in mind that only Building 1 & Building 2 has 6509 Cores but Building 3 & Building 4 has 3750 PoE Layer 3 switches.

How should i configure this network to make layer 2 loop free network or how should i design Spanning Tree Protocol for this network ?

Which spanning tree protocol type i should use?

Which switches should be the core switch.?

Should i keep same VTP domain among all buildings or each building should have separate VTP domain ?

What should be configured on Access Ports & Trunk ports/ Uplink to keep network safe ?

[devils_advocate]

Hi

If you chose this topology, you would need to enable a Routing protocol such as OSPF or EIGRP between each of the 4 buildings. Personally I would use EIGRP as I find it more simple to configure and troubleshoot (plus we have all Cisco) but if your skills are with OSPF or you need to extend the routing to non Cisco devices then use that

Spanning-tree is a Layer 2 protocol so its locally significant to each building. Personally I would be making the core switches in buildings 1 and 2 the root bridge for all vlans and using Rapid PVST. Assuming you have no other switches in buildings 3 and 4 then Spanning tree doesn't matter so much because they are a stack and there are no L2 trunks to other switches. I would still make the stacks the root bridges though because you may add some extra switches in the future and your Spanning tree is already configured properly.

Although it may look complicated, its a fairly simple and straight forward topology....

Routing protocol between the buildings, treat each building as a seperate LAN and follow best practice when configuring the switches.

[anis_cisco]

Hello Devils_Advocate  

Thanks for the reply, i understand your solution except you said

" Assuming you have no other switches in buildings 3 and 4 then Spanning  tree doesn't matter so much because they are a stack and there are no L2  trunks to other switches. I would still make the stacks the root bridges though because you may  add some extra switches in the future and your Spanning tree is already  configured properly. "

Can you explain it more please ?

Each building has the switches, building 1 & building 2 has 6509 core but rest buildings has 3750 PoE.

I am assuming to configure 3750 PoE as core for that building & then EIGRP.

Can we have multiple root bridges (STP) in a network ? so that in my scenario i can have  the core switch of each building as root bridge .

Regards,

[Jon Marshall]

Personally i would not reuse vlan IDs the way you have as it can lead to confusion ie. you have a vlan 10 in each building but using a different subnet per building. I would use different vlan IDs per building.  However it's not just a personal choice with your design because it also depends on where you plan to route the vlans for buildings 3 & 4.

Marvin has referenced using L2 etherchannels from the 3750s to the 6500s so in effect each 3750 building is treated as an access-layer in the traditional Cisco hieracrchy and you have a collapsed core/distro on the 6500s. Devils_advocate is referring to a L3 routed design where the 3750s route their local vlans and connect back to the 6500s with L3 etherchannels. This way treats each 3750 stack as a kind of collapsed access/distro layer and the 6500s purely as core. Both are perfectly valid design choices but they each have their own implications.

If you are planning to connect the 3750s with L2 trunks (Marvin's solution) then you cannot use the same vlan IDs because all the vlans would be routed off the 6500s so you could not have a vlan 10 SVI for each subnet ie. 10.1.10.x/10.3.10.x/10.4.10.x (well i suppose you could use secondary addressing but you really don't want to do this). So you would need unique vlan IDs per building so on the 6500s in building 1 you could configure unique SVIs for each vlan.

If you do want to route all vlans on the 6500s then you should renumber the vlans and you would make the 6500s in building 1 the STP root for all vlans on the 3750s.

If you are planning to route the local vlans in buildings 3 & 4 off their respective 3750s (which is the solution devils_advocate is referring to) then you would need to connect the 3750s back to the 6500s with L3 etherchannels. The 3750s would then become the STP root for the vlans in their respective building. With this solution you would also need to run a routing protocol between the 3750s and the 6500s. If you choose this solution then you would not technically need to renumber vlans but, as already stated, i would anyway.

Some further points -

1) The L2 solution is still loop free because you are running VSS on the 6500s. You should still run STP though.

2) The L3 solution where you route between all buildings has one major limitation in that you cannot have the same vlan (more specifically the same subnet) in multiple buildings. Your design for 4 builldings has unique subnets per building. If you know this will be the same for all new buildings then the L3 option will work fine but if you need to span a vlan (subnet) across multiple buildings later on then you won't be able to do this. The L2 option provides more flexibility in that respect. Ideally though you wouldn't want to be spanning a vlan across multiple sites. (the main one is where you have a common management vlan for all switches - see point 3).

It is worth spending some time investigating whether this could become an issue. It is unlikley for end clients in different buildings to need to be in the same subnet but servers can be a different matter.

3) Management of the 3750s. If you choose the L2 option ie. route the vlans off the 6500s then you can use a common management vlan between the 6500s in building 1 and the 3750 stacks. If you choose the L3 option then you can use loopbacks on the 3750s to manage them. The loopbacks would be advertised into the routing protocol so they could be reached from anywhere within your network.

4) Traffic flows. With the L2 option all traffic that is not in the same subnet will need to go via the 6500s in building 1 to be routed. So if a client in vlan 10 in building 3 needs to talk to a device in vlan 20 in building 3 it would need to be routed via the 6500s (note i have used your vlans numbers in this example but as already stated this L2 solution would need vlan renumbering).  With the L3 option the same traffic flow would be routed locally by the 3750 stack.

Whether this is an issue depends primarily on where your servers are based. If they are all in building 1 or 1 and 2 then the L2 option is fine because most of the traffic will need to be sent to the 6500s anyway. If however there are large data transfers betwween devices within the same building it may make more sense to route that locally on the 3750s. If you are scaling up to 16 buildings then it is definitely a consideration you need to take into account.

Jon

[anis_cisco]

Hello Jon.marshall,

Thank you so much for such a deep explaination & thoughts appreciated.

Purpose of using same Vlan ID's for to keep standard like Admin Department in all buildings would be Vlan 10, Students in all buildings would be Vlan 20 like that.

Our goal is to consider each building as an isolated network then other building, meaning every building should have there own Vlans & subnet, intervlan routing of each building should be routed through its own core switch (e.g. for building 1 core is 6509, for building 3,4 core is 3750) Incase if building 1 wants to communicate with building 3 that should be L3 communication like OSPF or EIGRP.

So i will go with Devils_Advocate solution to configure Layer 3 Etherchannels between core of each buildings plus EIGRP routing protocol on it. Each building core will perform intervlan routing for its own building.

Regarding STP, I will manually configure the core switch of each building as Root Bridge for its own building to avoid layer 2 looping within a building.

Till here i am fine & i will finalize the design as above plus as suggested by Devils_Advocate.

Thanks Devils_Advocate, your name remind me one of my favorite movie name Devils_Advocate

Regarding Servers, all are residing in Building 1. Customer has two internet links from two different ISP (1 internet line for Students & 1 for Teachers) , they have two ISA Proxy servers (1 internet line for each ISA Server), customer has 1 Exchange, 1 DNS Active Directory, File Server & few more Windows based server. Total number of servers are around 20.

Where should i deploy the ISA server in network ? Should i connect both to the core switch directly ?

How should i route the traffic of students subnet to 1st ISA & for teachers subnet to 2nd ISA ? Should i use Route Maps or Policy Base Routing to route the traffic of specific subnets to specific gateway ??

Regards,

[Jon Marshall]

Where should i deploy the ISA server in network ? Should i connect both to the core switch directly ?

It depends on where you are connecting the rest of the servers.In the traditional cisco design the core is really only for interconnecting network devices and it is left to switch (L2 or L3) between devices so ideally you would want a dedicated pair of switches for the servers which then connect to the core switches. But this often comes down to cost and if you have the spare ports on the 6500s you can use them.

However It's not clear from your diagram where the internet connectiity is and how it is handled unless it is the ASA -> MPLS bit but that only shows one ASA with one connection but you are referring to 2 internet connections. Perhaps you could clarify ?

I ask because it may be there is a more logical place to connect the ISA servers that is not shown on your design.

How should i route the traffic of students subnet to 1st ISA & for teachers subnet to 2nd ISA ? Should i use Route Maps or Policy Base Routing to route the traffic of specific subnets to specific gateway ??

You can use policy routing but i although i haven't used ISA if it is a proxy server this can be defined within an AD group policy and the correct proxy server can be set in the users browser. So the students subnets would get one proxy server and the teacher subnet would get the other proxy server. Note that i am not a windows person but i am assuming that you can define a group policy based on subnet being used.

At my last place we used group policy to set the proxy server in the browser. The desktop was locked down so this couldn't be changed. Even with this though you still need to configure your firewall to only allow http/https etc. from the proxy server address to be allowed out to the internet otherwise a user could bring in their own laptop and simply bypass the proxy server.

Jon

[Jon Marshall]

One thing i forgot to ask. In addition to query about internet setup are you hosting any services that are accessed by users on the internet eg. web/ftp etc ?

Jon