Confused: Connecting 2 buildings with 3560 Switches

mconway · ‎12-22-2011

Here goes ... the problem:

I will have 2 physically seperated buildings on the same lot connected via fiber optics at each end are 3560 switches that will do routing. Each building will be on it's own subnet.

Building A is 10.0.0.0/24 VLAN 1 (Which currently has (4) 3560 switches daisy chained together). I would like to move all the devices connected to these switches to a diffrent VLAN. This will be where the majority of the servers will remain.

Building B will be 10.1.0.0/24 VLAN 10.

My plan is to install a Domain Controller on 10.1.0.0/24.

How do I configure the switches to route correctly between themselves and have 10.1.0.0/24 route internet traffic through a PIX on 10.0.0.0/24 ip 10.0.0.1/24. I'm leary of making changes to the existing 10.0.0.0/24 because of it being live.

darren.g · ‎12-22-2011

Michael Conway wrote:

Here goes ... the problem:

I will have 2 physically seperated buildings on the same lot connected via fiber optics at each end are 3560 switches that will do routing. Each building will be on it's own subnet.

Building A is 10.0.0.0/24 VLAN 1 (Which currently has (4) 3560 switches daisy chained together). I would like to move all the devices connected to these switches to a diffrent VLAN. This will be where the majority of the servers will remain.

Building B will be 10.1.0.0/24 VLAN 10.

My plan is to install a Domain Controller on 10.1.0.0/24.

How do I configure the switches to route correctly between themselves and have 10.1.0.0/24 route internet traffic through a PIX on 10.0.0.0/24 ip 10.0.0.1/24. I'm leary of making changes to the existing 10.0.0.0/24 because of it being live.

Easiest way would be to configure an the interface you use to link the 3560's as a routed port, assign it a /30 address and then just static route between the switches.

Example: Assuming you use g1/1/1 on the 3560 at each end fo run the fibre link, VLAN 10 for 10.0.0.0/24 and VLAN11 for 10.1.0.0/24

int g1/1/1

no switchport

media-type sfp

ip address 10.100.1.1 255.255.255.252

no shut

exit

int vlan10

ip address 10.0.0.254 255.255.255.0

no shut

ip route 0.0.0.0 0.0.0.0 10.0.0.1

ip route 10.1.0.0 255.255.255.0 10.100.1.2

int g1/1/1

no switchport

media-type sfp

ip address 10.100.1.2 255.255.255.252

no shut

exit

int vlan11

ip address 10.1.0.254 255.255.255.0

no shut

ip route 0.0.0.0 0.0.0.0 10.100.1.1

This tells the layer 3 process of switch 2, which houses the 10.1.0.0/24 subnet (.254 assumed as default router) to send ALL traffic to the other end of the point-to-point link between the two switches - and then lets switch a, which houses 10.0.0.0/24 decide how to forward it to the PIX (default route sends all unknown traffic to 10.0.0.1).

Switch b doesn;t need more than a single static route because you have nothing complex on it, and no multiple exit points - it's a simple "send everything which isn't in my subnet HERE" situation.

Cheers.

mconway · ‎12-23-2011

Thank you for your response darren it is very helpful. Though i do have questions..............

switch a - 10.10.0.0/24 (Is this suppose to be 10.0.0.0/24 ?)

I'm not familiar with int g1/1/1. On the switches I have the SFP ports are GigabitEthernet 0/49 - 0/52 ....

int VLAN 10 you are assigning 10.0.0.254. This will be the default gateway for all client pc's / servers on 10.0.0.0/24 correct? Currently the default gateway for the clients is 10.0.0.1/24 which is the inside interface of the PIX. I will NOT need to setup up a static routable connection to the PIX correct? I just leave as is which is the PIX is just connected to an access port and I point the unknown traffic to ip route 0.0.0.0 0.0.0.0 10.0.0.1?

In my situation I currently have 4 switches daisy chained together and trunked with dot1q in my orginal network. When I enable ip routing do i need to do this on just the switches with the physical 10.100.1.0 / 255.255.255.252 and PIX connetions?

What kind of disruption moving devices from VLAN 1 to VLAN 10 might happen.. IE will there be downtime? What about enabling ip routing. I have the switches already configured with STP, Pruning, and clustering.

darren.g · ‎01-02-2012

Michael Conway wrote:

Thank you for your response darren it is very helpful. Though i do have questions..............

switch a - 10.10.0.0/24 (Is this suppose to be 10.0.0.0/24 ?)

I'm not familiar with int g1/1/1. On the switches I have the SFP ports are GigabitEthernet 0/49 - 0/52 ....

int VLAN 10 you are assigning 10.0.0.254. This will be the default gateway for all client pc's / servers on 10.0.0.0/24 correct? Currently the default gateway for the clients is 10.0.0.1/24 which is the inside interface of the PIX. I will NOT need to setup up a static routable connection to the PIX correct? I just leave as is which is the PIX is just connected to an access port and I point the unknown traffic to ip route 0.0.0.0 0.0.0.0 10.0.0.1?

In my situation I currently have 4 switches daisy chained together and trunked with dot1q in my orginal network. When I enable ip routing do i need to do this on just the switches with the physical 10.100.1.0 / 255.255.255.252 and PIX connetions?

What kind of disruption moving devices from VLAN 1 to VLAN 10 might happen.. IE will there be downtime? What about enabling ip routing. I have the switches already configured with STP, Pruning, and clustering.

Yeah, the IP address for switch A is whatver you have on your end - 10.0.0.0/24 - just use whatever you have configured. My stuff was meant as an example, not to be directly cut and paste. :-)

As far as the interface designations go, use your ones - I'm not familiar with the 3560, so I just used a designation from one of my 3750-X's - your port designations will be different - once again, my stuff was intended as an example, not a cut and paste job.

By configuring an SVI in VLAN10 and assigning it an IP address, you create the router for the entire system - this will be the default router for the 10.0.0.0/24 and all PC's assigned to it - but you need to tell it how to get traffic for which it doesn't have a route (I.E. default route) to the PIX - so you DO need to setup a statis route to the PIX - which is what the line in my example

ip route 0.0.0.0 0.0.0.0 10.0.0.1

does. The PIX will also need to have an inbound route for the INSIDE networks (I assume it already has a default route for outbound trafrfic to the Internet) telling it how to get traffic for the inside networks (10.0.0.0/24 & 10.1.0.0/24) - you would make these routes point to 10.0.0.254 - so on the PIX you would need statements like this (depends on software version and interface name, so might not be exact - I've assumed your inside interface is called "inside")

route inside 10.0.0.0 255.255.255.0 10.0.0.254

route inside 10.1.0.0 255.255.255.0 10.0.0.254

If your switches are trunked together and the VLAN ID's match, you only have to apply an SVI to one of them - preferably the one with the PIX conencted, but it doesn;t have to be. You WILL have to make sure you create the VLAN (VLAN10) on all the interconnected switches, and assure that the dot1q trunks allow this VLAN, *and* that all access prots are in this access VLAN. You could, if you chose, put in place a HSRP or VRRP configuration across two switches to allow for the possibility that your routing switch fails - but that's up to you.

Moving from VLAN1 to VLAN10 will cause an outage, but provided you have your SVI setup and your trunks all OK first, it should only be for about 30 seconds per node (assuming you cutover one node at a time), although once you move your PIX into VLAN10 nothing on VLAN1 will work anymore until it gets moved to VLAN10 and can talk to the PIX again. You'll also, obviously, have to change your default gateways on your nodes (from the PIX IP address of 10.0.0.1 to the SVI IP address of 10.0.0.254), and if you're using DHCP you might have to either modify your DHCP configuration or apply an ip helper command (if you're not using the switches as a DHCP server) to relay the DHCP requests to your DHCP server.

Phew. Long winded for the new year!

Hope this helps. Please rate posts/mark answered if happy.

flokki123 · ‎01-03-2012

hi darren,

just read your answer and it makes total sense.

would like to know if it also would work, if michael would create on each site the non-existing vlan(vlan10 building A and vlan 1 building B) make a trunk connection between the 2 switches.

so the switches would route between the 2 vlans and the trunk connection would connect the vlans between each other.

thanks.

florian

darren.g · ‎01-03-2012

flokki123 wrote:

hi darren,

just read your answer and it makes total sense.

would like to know if it also would work, if michael would create on each site the non-existing vlan(vlan10 building A and vlan 1 building B) make a trunk connection between the 2 switches.

so the switches would route between the 2 vlans and the trunk connection would connect the vlans between each other.

thanks.

florian

Florian.

Yes, you could link the switches via a trunk over the fibre and have the SVI's on one switch as a primary (the main one in building A, for example) - there's no real difference - but the original post pretty much stated the two buildings were effectively separate entities, so I kept them that way. One advantage of using a trunk would be the ability to port-channel the intra-building link should additional bandwidth be required without having to reconfigure routing etc - my original suggestion assumed a single port scenario linking the buildings - if you wanted to run more than one fibre pair between buildings and get more intra-building bandwidth you could connect the ports as a port-channel trunk and get it.

Cheers.

flokki123 · ‎01-04-2012

hi darren,

thanks for the explanation.

florian

mconway · ‎01-03-2012

THANK YOU SO VERY MUCH!

My understanding has really gotten really clearer with your help. Yep I was doing a cut and paste .

I aleady have a VTP domain setup so that will take care of propagating the VLAN information once I setup the VLAN on the one switch it will than broadcast the VLAN information to the other switches.

Changing the default gateway on all the nodes will be a pain due to the statically configurations. The Exchange server will not be happy until i reboot it for sure with a change to its default gateway. I have a Windows 2008 R2 DC that does DHCP which is simple to change the default gateway for the dhcp clients. In the new building with it's own DC / DHCP 2008 R2 Server I will configure the default gateway to be that of the SVI of the new switch in the new building.

QUESTION:

On the routable connection between building can I configure LACP with the 2 physical fiber connections that I will have avaible? This would give me a 2 Gbps connection with some failover if one line were to go down.

darren.g · ‎01-03-2012

Michael Conway wrote:

THANK YOU SO VERY MUCH!

My understanding has really gotten really clearer with your help. Yep I was doing a cut and paste .

I aleady have a VTP domain setup so that will take care of propagating the VLAN information once I setup the VLAN on the one switch it will than broadcast the VLAN information to the other switches.

Changing the default gateway on all the nodes will be a pain due to the statically configurations. The Exchange server will not be happy until i reboot it for sure with a change to its default gateway. I have a Windows 2008 R2 DC that does DHCP which is simple to change the default gateway for the dhcp clients. In the new building with it's own DC / DHCP 2008 R2 Server I will configure the default gateway to be that of the SVI of the new switch in the new building.

QUESTION:

On the routable connection between building can I configure LACP with the 2 physical fiber connections that I will have avaible? This would give me a 2 Gbps connection with some failover if one line were to go down.

Michael.

See the answer I just wrote to Florian - the way I originally suggested you could not run an port-channel and LACP, since I assigned a single IP address to the physical interface.

If you wanted to run multiple fibre links and get 2 gig bandwidth, you would need to do similar to this - again, this is *not* a cut-and-paste - you'll need to massage it to suit your environment

Switch A, Building A

interface port-channel 1

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,10,11

switchport mode trunk

no shut

interface g0/49

switchport mode trunk

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,10,11

channel-group 1 mode active

no shut

interface g0/50

switchport mode trunk

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,10,11

channel-group 1 mode active

no shut

vlan 10

name Building_A

vlan 11

name Building_B

interface vlan 10

ip address 10.0.0.254 255.255.255.0

interface vlan 11

ip address 10.1.0.254 255.255.255.0

ip route 0.0.0.0 0.0.0.0 10.0.0.1

Switch B, building B

interface port-channel 1

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,10,11

switchport mode trunk

no shut

interface g0/49

switchport mode trunk

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,10,11

channel-group 1 mode active

no shut

interface g0/50

switchport mode trunk

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,10,11

channel-group 1 mode active

no shut

vlan 10

name Building_A

vlan 11

name Building_B

Put all your ports except the links in building A into access mode, VLAN 10 and all your ports in building B into access mode, VLAN 11

You should then need nothing of routing on switch B in building B - it should all flow over the trunk to the router on switch A in building A.

Of course, if you ever lose your fibre, then building B will effectively be completely isolated - you could move the SVI for VLAN 11 to it, but if you do that and lose the fibre you're not goint to get anywhere anyway.

You could also run HSRP on both switches (create an SVI for VLAN 10 and VLAN 11 on both of them) and have routing failover in case of issues at layer 3 - that might be more complex than you need, though.

Cheers

BOGDAN OVIDIU STANCIU · ‎01-05-2012

Hello,

Darren, why Michal cannot make etherchannel Layer3 connections between those 2 switches?

He just have to change the layer3 settings from physical interfaces to the port-channel interface. After this he can add up to 8 physical interfaces to this port-channel, having redundancy and load-balance!

So, Switch A - 10.10.0.0/24>

int g1/1/1

no switchport

no ip address

channel-group 1 mode active

no shut

exit

!

int g1/1/2 #### the new interface available####

no switchport

no ip address

channel-group 1 mode active

no shut

exit

!

interface port-channel 1

no switchport

ip address 10.100.1.1 255.255.255.252

no shut

exit

SWITCH B - 10.10.1.0/24>

int g1/1/1

no switchport

no ip address

channel-group 1 mode active

no shut

exit

!

int g1/1/2 ##### the new interface on SWITCH B#

no switchport

no ip address

channel-group 1 mode active

no shut

exit

!

interface port-channel 1

ip address 10.100.1.2 255.255.255.252

no shut

exit

!

The routing is the same!

Michal can use channel-group mode active if he wants LACP, but I recommend to use "mode on". In this case there is no LACP or PAGP used.

I hope it helps,

Bogdan

darren.g · ‎01-05-2012

BOGDAN OVIDIU STANCIU wrote:

Hello,

Darren, why Michal cannot make etherchannel Layer3 connections between those 2 switches?

He just have to change the layer3 settings from physical interfaces to the port-channel interface. After this he can add up to 8 physical interfaces to this port-channel, having redundancy and load-balance!

Bogdan.

You could, I guess - but why add an additional layer 3 process to the ether channel if you don't have to? Also, doing it with a layer 3 port channel instead of a layer 2 port channel adds complexity, if not removes completely, the possibility of doing HSRP between the two switches (not that this is really necessary, but it's a "nice to have".).

BTW, from memory Cisco recommends using "mode on" only if you absolutely have to (I.E. there's a device at the other end which won't negotiate) as it removes some of the flexibility from port channels. As I said, from memory - I always use "mode active" on both ends, or at worst "mode active' on one end and "mode passive' on the other, if I'm using two Cisco devices which support LACP.

For every question, there's an alternate solution - my original reply didn't take in the consideration that maybe Michael had a requirement for more than one link between buildings, my second gives one option to achieve that - you may do it differently, and someone else might do it differently still - as I said a couple of times, I wasn't laying out a cut-and-paste, guaranteed to work solution - suggestions posted by me are based on my understandings, and will usually have to be tweaked to work anywhere except in my head (or a network I put together from scratch!). :-)

Cheers.

BOGDAN OVIDIU STANCIU · ‎01-06-2012

You are right! :-)

Ofcourse is easy and faster to use a Layer 2 EtherChannel and Michal can put additional fibers into it.

I hope Michal is having a wide picture now of the many possibilities that he has.:)

All the best,

Bogdan

mconway · ‎01-06-2012

Yep, I am getting a wide picture of the many possiblities available.

I have been literally learning from the seat of my pants with this cisco gear and LAN technology. The fiber connection has yet to be pulled between buildings (thinking mid January) so in the mean time i have a small equipment rack setup on my desk to learn with and how best to design the LAN

Many thanks to everyone.

I never would have quessed how helpful posting on this forum would be.

mconway · ‎01-06-2012

In the past when I have configured an LACP link between a server and a switch I could see the connection speed on the virtual adapter that is created when teaming on the server. How do I check the speed with 2 switches connected via a LACP Trunk?

In darren's example he creates the port-channel with the encapsulation dot1q then also configures each interface with dot1q encapsulation that will be in the port-channel. Is there a reason for this? Seems double redundant? Is this for failover?

BOGDAN OVIDIU STANCIU · ‎01-06-2012

Hi,

The speed of the Port-Channel (Etherchannel) Interface is given by the speed of the physical interfaces bundled. If you have 2 ports of 1 Gigabit in one port-channel you will have a 2 gigabit etherchannel interface.:)

To see the bandwidth you can use:

SW# sh int po1

Port-channel1 is up, line protocol is up (connected)

Hardware is EtherChannel, address is 0019.5528.c990 (bia 0019.5528.c990)

Description: UCS PROD

MTU 9000 bytes, BW 2000000 Kbit, DLY 10 usec

To see information about the Port-channel use:

# sh etherchannel summary

# sh etherchannel 1 summary

or use the inline help with "?" like: sh ether ?

The port-channel (etherchannel) interface is a virtual one. It inherits the configuration of physical interfaces (physical switch ports).

So, in order to create a port-channel interface, you have to use the same configuration parameters (encapsulations, speed, vlans, allowed vlans, etc) for all physical interfaces (ports) that will be part of that port-channel.

That's why Darren used encapsulation dot1q on both physical interfaces.

The redundancy is only at the physical interfaces level. When one port fails (from different reasons) the other(s) will carry the traffic.

I hope I've answered your questions:) .

Bogdan