Adding Metro E to connect remote site site

Unanswered Question
Jul 8th, 2009

We currently have a routed connection to a site in a city accross the state.

The DR site is configured with a totally seperate VLAN config.

For example everything in the Main site is 10.100.x.x\24,

10.100.1.x\24 servers

10.100.2.x\24 printers

10.100.3.x\24 workstations

Remote site is 10.200.x.x\24

10.200.1.x\24 servers

10.200.2.x\24 printers

10.200.3.x\24 workstations

When we move from routed to Metro E to connect this site, is a trunk to the other site the way to go to connect the other site VLANs to the main site VLANS?

We have 5-060 VLANs in the main site and 20-30 VLANS in the remote site.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (12 ratings)
Loading.
Edison Ortiz Wed, 07/08/2009 - 06:47

The provider will give you a Layer2 handoff with a customer Vlan.

On the ISP facing interface, you can configure a trunk with the customer Vlan being the native Vlan.

For instance; customer Vlan 100

interface fx/x

switchport trun en do

switchport trun na vl 100

switchport mode trunk

switchport trun all vl 100,101

The 101 Vlan can be the Vlan where you create the p-t-p link for this MetroE so in your switches create Vlan 101 on L2 and L3 and assign the L3 Address as /30 subnet.

For instance:

Main Site

interface vlan 101

ip add 10.101.1.1 255.255.255.252

Remote Site

interface vlan 101

ip add 10.101.1.2 255.255.255.252

Then you can configure your routing protocol of choice and advertise the subnets from each location.

HTH,

__

Edison.

wilson_1234_2 Wed, 07/08/2009 - 08:31

Thanks Edison,

At both ends, we are using OSPF locally to route the VLANs to all devices.

Also we are bridging several VLANs across the routed links.

I think I see where you are going with connecting the Layer 3 links across the span, but will the provider be handing off a trunk to me?

Also, what about the layer 2 VLANs?

We have an application that requires the failover server to be in the same subnet as the Primary.

This would be easy with just a layer 2 connection to the remote end.

Edison Ortiz Wed, 07/08/2009 - 08:51

The provider will have a dot1q-tunnel on their access port with a customer vlan as the native vlan.

You can configure 802.1q on your switches to carry multiple Vlans.

If you want to extend some L2 information over the MetroE, just allow those Vlans on the trunk. Make sure to perform manual pruning with the command I entered before 'switchport trunk all vl x,x,x'.

HTH,

__

Edison.

wilson_1234_2 Wed, 07/08/2009 - 12:18

Oh, I see Edison, that cleared it up.

Basically I am routing over the layer 2 network and trunking the VLANs needed in both sites.

Am I loosing any benefit of the Metro E spanning the two sites by doing it this way?

I guess the only way to trunk everything is to change the addressing scheme in one of the sites correct?

Edison Ortiz Wed, 07/08/2009 - 12:22

Think of MetroE as interconnecting 2 switches together in your LAN.

The only caveat is the customer Vlan that must be untagged towards the carrier. Everything else is fair game, including CDP :) Nice to see your remote switches as directly connected CDP neighbors.

___

Edison.

wilson_1234_2 Fri, 07/10/2009 - 12:15

Edison,

In my scenario, since I am already using OSPF in each respective site, and they are configured in the same area, once the trunk is configured, OSPF should form a neighbor adjacentcy in both sites across VLAN 101 is this correct?

And VLAN 101 is the path carrying all routed traffic across the link?

mbroberson1 Tue, 07/14/2009 - 18:25

Hi Edison,

Great post here. My company is moving to ME for several of our remote sites. Could you please describe the setup if you are using routers (not switches) at your remote sites for the CE router where the provider will handoff? We will be doing a ME "hub and spoke" type of setup.

Thanks,

Brandon

Edison Ortiz Wed, 07/15/2009 - 05:14

ISP facing interface at the CE:

interface gx/x

no ip address

interface gx/x.x

encapsulation dot1q [Vlan_Given_By_ISP] native

interface gx/x.x

encapsulation dot1q [Your_Vlan]

ip address x.x.x.x y.y.y.y

interface gx/x.x

encapsulation dot1q [Your_Vlan]

ip address x.x.x.x y.y.y.y

etc.

__

Edison.

mbroberson1 Wed, 07/15/2009 - 05:21

Hi Edison,

Thanks for this response. I am assuming this will be the HUB end. Can you give an example of what the remote site router config may look like? I think I know, but would like your input. Also will the [Your_Vlan] part be the vlan I come up with? For example if I say vlan 10 for site 1 then I will assign that to my subinterface on the HUB router and on the remote router use the same dot1q on the interface facing the provider?

Thanks,

Brandon

Edison Ortiz Wed, 07/15/2009 - 05:26

The Hub will have all the subinterfaces and IP subnets for the remotes in addition to the native subinterface for the ISP. The Remotes will have the native subinterface for the ISP in addition to the subinterface for the Hub with its corresponding subnet.

When I refer to the native subinterface for the ISP, it means you aren't tagging the Vlan on that subinterface while using the 'native' keyword on the encapsulation dot1q command.

___

Edison.

mbroberson1 Wed, 07/15/2009 - 05:28

That makes sense! Man you have been more that helpful Edison. I will rate your reply.

Thanks,

Brandon

mbroberson1 Wed, 07/15/2009 - 05:39

Edison,

If you are doing "router-on-a-stick" at the remotes will you just have to just make sure your "routed vlan" for the point-to-point" link is different and unique, not used in the LAN portion of your remote site? I hope I'm making sense.

Thanks,

Brandon

mbroberson1 Wed, 07/15/2009 - 05:41

What I meant by that last post is you need to make sure the native the ISP gives you and your native at the remote is a different vlan number. Right?

-Brandon

Edison Ortiz Wed, 07/15/2009 - 05:42

You can re-use Vlans on this scenario as the router will break the tag at ingress from the MetroE network and send it routed back to the internal network. At the internal network the switch connecting to this router can apply another Vlan tag or leave the incoming traffic from the router untagged.

Your Vlan allocation concern would be if the router is also doing 'router-on-a-stick' internally.

__

Edison.

mbroberson1 Wed, 07/15/2009 - 05:51

Hi Edison,

So say the below config is you current router-on-a-stick configuration for a remote site (before ME implementation). And the provider gives you a vlan of say 200 to use. How could you arrange this config for the remote?

interface FastEthernet0/0

no ip address

speed 100

full-duplex

!

interface FastEthernet0/0.1

encapsulation dot1Q 1 native

ip address 10.10.1.208 255.255.255.0

!

interface FastEthernet0/0.2

description DHCP AP's

encapsulation dot1Q 2

ip address 10.10.2.1 255.255.255.0

ip helper-address 10.40.2.3

ip helper-address 10.40.2.4

!

interface FastEthernet0/0.3

description fhmi$access

encapsulation dot1Q 3

ip address 10.10.3.1 255.255.255.0

ip helper-address 10.40.2.3

ip helper-address 10.40.2.4

!

interface FastEthernet0/0.6

description Static IPs

encapsulation dot1Q 6

ip address 10.10.6.1 255.255.255.0

!

interface FastEthernet0/0.7

description PBX

encapsulation dot1Q 7

ip address 10.10.7.1 255.255.255.0

Thanks,

Brandon

Edison Ortiz Wed, 07/15/2009 - 05:56

That's your internal facing interface. This router will need a ISP facing interface.

On the ISP facing interface, say F0/1

interface FastEthernet0/1

no ip address

speed 100

full-duplex

!

interface FastEthernet0/1.200

description ISP Native Vlan

encapsulation dot1Q 200 native

interface FastEthernet0/1.xxx

description point-to-point subnet for Hub

encapsulation dot1Q xxx

ip address x.x.x.x y.y.y.y

mbroberson1 Wed, 07/15/2009 - 06:02

Hi Edison,

You're correct...still waking up I guess...;-).

Thanks,

Brandon

wilson_1234_2 Wed, 07/15/2009 - 10:54

Edison,

In my scenario (from the top of post), I have two sites, with two different VTP domains.

Would you suggest that I add the reote site to the VTP domain in the main site, or leave it seperate?

I will be routing all but two or three VLANs (with the possibility of more later).

Edison Ortiz Wed, 07/15/2009 - 11:06

Up to you. If you want to add/modify/delete Vlans for remote site from a HQ VTP server, then configure the same VTP domain across.

I personally wouldn't do that. Have the remotes have a different VTP domain (Server/Client) from the HQ.

Edison Ortiz Thu, 07/16/2009 - 05:58

It provides some kind of Vlan management security.

You don't want a mistake on Vlan creation/modification/deletion to get propagated to other switches.

We usually recommend running VTP transparent in a switched network. Note: This has nothing to do with MetroE but just a general concept.

HTH,

__

Edison.

wilson_1234_2 Fri, 07/17/2009 - 11:11

I am not sure what you are saying about the VTP,

So, with two 6509 switches at the core network with several 3550 access switches,

All would be transparent?

Currently we have the two 6509s running VTP, one is server the other is client and the rest are transparent.

mbroberson1 Fri, 07/17/2009 - 10:52

Hi Edison,

Just curious, but could you give a brief config or explanation of the configuration if you had a switch (6500) at the hub end instead of routers?

Thanks,

Brandon

Edison Ortiz Fri, 07/17/2009 - 10:56

Much easier;

interface gx/x

description MetroE connection

switchport trunk encapsulation dot1q

switchport mode trunk

switchport trunk native vlan [Metro_Vlan]

switchport trunk allow vlan [Metro_Vlan],[Common Vlans to be used for Remotes....]

interface vlan [remote 1]

description Site 1

ip address x.x.x.x y.y.y.y

interface vlan [remote 2]

description Site 2

ip address x.x.x.x y.y.y.y

mbroberson1 Fri, 07/17/2009 - 12:25

Hi Edison,

So the HUB is a switch and the remotes are routers.

Would the interface on the remote routers sub interface pointing to the ISP (hand-off) reference dot1q of the vlan at the HUB?

Such as:

HUB switch:

interface gx/x

description MetroE connection

switchport trunk encapsulation dot1q

switchport mode trunk

switchport trunk native vlan 500

switchport trunk allow vlan 500,150,160

interface vlan 150

description Site 1

ip address 10.150.255.1 255.255.255.252

interface vlan 160

description Site 2

ip address 10.160.255.1 255.255.255.252

Remote site routers:

[remote 1]

interface FastEthernet0/1

no ip address

speed 100

full-duplex

!

interface FastEthernet0/1.1

description point-to-point subnet for Hub

encapsulation dot1Q 150

ip address 10.150.255.2 255.255.255.252

!

interface FastEthernet0/1.200

description ISP Native Vlan

encapsulation dot1Q 500 native

!

interface FastEthernet0/0

no ip address

speed 100

full-duplex

!

interface FastEthernet0/0.1

encapsulation dot1Q 1 native

ip address 10.20.1.211 255.255.255.0

[remote 2]

interface FastEthernet0/1

no ip address

speed 100

full-duplex

!

interface FastEthernet0/1.1

description point-to-point subnet for Hub

encapsulation dot1Q 160

ip address 10.160.255.2 255.255.255.252

!

interface FastEthernet0/1.200

description ISP Native Vlan

encapsulation dot1Q 500 native

!

interface FastEthernet0/0

no ip address

speed 100

full-duplex

!

interface FastEthernet0/0.1

encapsulation dot1Q 1 native

ip address 10.30.1.207 255.255.255.0

Thanks,

Brandon

mbroberson1 Fri, 07/17/2009 - 12:35

Awesome,

And if I did go with the switch method at the HUB and say my remotes were 8MB ME connections to do QoS would at the hub end would I configure this under the VLAN sub-interfaces? I know a switch typically isn't as smart as a router (with QOS, shaping, etc...) so what would you advise for someone with these concerns.

Any recommendations?

Kind Regards,

Brandon

Edison Ortiz Fri, 07/17/2009 - 12:42

No shaping available on regular line cards on a 6500. You need a SIP with Ethernet SPAs. On the SPA, you would do shaping like a regular IOS router and apply the service-policy on the main interface.

HTH,

__

Edison.

mbroberson1 Fri, 07/17/2009 - 12:45

But with the router method at both the hub and remote ends it basically works like Point-to-point circuits...sort of? Where you can do the QoS service-policy and apply to oustside interface facing ISP hand-off on each side?

Thanks,

Brandon

Edison Ortiz Fri, 07/17/2009 - 13:07

It's a point-to-multipoint connection as your router is connected to a switched network so creating the subinterfaces in the router allows you to carry additional Vlans and L3 subnets - so don't use the term point-to-point :)

You are correct on the QoS and I want to expand a bit, if you go with the SIP/SPA option on the 6500, you will need to create subinterfaces on the Ethernet SPA for QoS shaping support.

HTH,

__

Edison.

wilson_1234_2 Mon, 07/20/2009 - 08:00

Edison, another couple of questions on the Metro E connection using routers:

In the examples shown in this post, the Subinterfaces on the router would be for the VLANs to be trunked from one end of the Metro E connection to the other, but in my scenario, I would only be trunking a few VLANs and routing the others(keeping the VLANs and IP Addressing seperate).

HQ_VLANs.......................DR_VLANs

6509--->7206-->Metro_E-->7206-->3750

If I were to combine the two examples here (switch and router), and since my switches are layer three switches on both ends running OSPF, would it look like the below?

HQ router:

On the ISP facing interface, say F0/1

interface FastEthernet0/1

no ip address

speed 100

full-duplex

!

interface FastEthernet0/1.xxx

description WAN point-to-point to ISP for routed subnets

encapsulation dot1Q xxx

ip add 10.100.1.1 255.255.255.252

!

interface FastEthernet0/2.xxx

encapsulation dot1q [Your_Trunked_Vlan]

!

interface FastEthernet0/3.xxx

encapsulation dot1q [Your_Trunked_Vlan]

Internal Facing Interface

interface FastEthernet0/0

no ip address

speed 100

full-duplex

!

interface FastEthernet0/0.1

description LAN point-to-point switch for routed subnets

encapsulation dot1Q 1 native

ip add 10.101.1.1 255.255.255.252

!

interface FastEthernet0/0.2

description Trunked to DR

encapsulation dot1q [Your_Trunked_Vlan]

!

interface FastEthernet0/0.3

description Trunked to DR

encapsulation dot1q [Your_Trunked_Vlan]

router ospf 1

network 10.100.1.0 0.0.0.3 area 0

network 10.101.1.0 0.0.0.3 area 0

HQ Switch:

interface FastEthernet0/0

no ip address

speed 100

full-duplex

!

interface FastEthernet0/0.1

description LAN point-to-point for routed subnets

encapsulation dot1Q 1 native

ip add 10.101.1.1 255.255.255.252

!

interface FastEthernet0/0.2

description Trunked to DR

encapsulation dot1q [Your_Trunked_Vlan]

!

interface FastEthernet0/0.3

description Trunked to DR

encapsulation dot1q [Your_Trunked_Vlan]

router ospf 1

network 10.101.1.0 0.0.0.3 area 0

network [VLANs to be routed] area 0

Also, if there was an option between a Metro E connection or a DS3,is there any difference in performance or amount of traffic going across the Metro E link configured per the above, as opposed to using the DS3 and bridging the desired VLAN traffic and routing the rest?

Basically, it is doing the same thing, but in reverse. Is there a performance benefit with using the Metro E connection (assuming the same bandwidth) over the DS3 with bridging configured?

The broadcast traffic would be the same correct?

Edison Ortiz Mon, 07/20/2009 - 08:18

Richard,

Per your requirements of 'trunking some Vlans while routing others' you need a switch as the CE device instead of a router.

While you can use the subinterface method to tag Vlans on the interface, this method won't extend the L2 information via the router. If you want to extend L2 information via the router, you need to use Integrated Routing and Bridging (IRB) which can complicate things. This approach is much cleaner with a switch design.

As the differences between a DS3 and a Metro-E, hmm just about 65Mbps :) Kidding aside, a Metro-E delivers high bandwidth at a much cheaper price than DS3. DS3 is often seen when interconnecting sites that aren't close in proximity while Metro as the name implies is for same city interconnects.

wilson_1234_2 Mon, 07/20/2009 - 08:46

Thanks Edison,

"While you can use the subinterface method to tag Vlans on the interface, this method won't extend the L2 information via the router."

Is this because of what you mentioned earlier, in the post that the prider will strip VLAN tags off on their trunk to the remote side?

I was thinking along the lines of comparison of the two technlogies and performance, if both had the same bandwidth.

We currently are using x2 point to point DS3s in a remote city, but have the option of gouing to a Metro E.

I was wondering if the Bridging (usually frowned upon) the VLANs via the router, rather than the Metro E is basically the same performance wise.

Doesn't the router add the layer two headers in the routed packet during rouing to the remote side?

This would be less efficient than the switch doing the trunking correct?

I was also thinking that the router would allow more options with QoS than the switch would.

Edison Ortiz Mon, 07/20/2009 - 09:02

Yes, the router has the ability to send and receive tag 802.1q packets but it won't extend the L2 domain end-to-end with subinterfaces.

There are similar technologies but the hardware required for each technology differs. For DS3, the CE must be a router or a high end modular switch while for Metro, the CE can be a router or any switch.

Bridging on a router can never be compared to L2 switching on a regular switch. It's night and day. Stay away from bridging on a router if possible.

Doesn't the router add the layer two headers in the routed packet during rouing to the remote side?

You lost me there. The router will add its own MAC address on the L2 header when sending the packet to the Metro-E network.

I was also thinking that the router would allow more options with QoS than the switch would.

This is when you get a 6500 with a SIP and Ethernet SPAs.

wilson_1234_2 Mon, 07/20/2009 - 10:05

For bridging layer 2 across a routed link:

Doesn't the router add the layer two headers in the routed packet during routing to the remote side?

Edison Ortiz Mon, 07/20/2009 - 10:17

If you are bridging on a router, you don't have a routed link, you will have the L3 information under the BVI hence my confusion on your question.

The router's physical port will be strictly L2 ports and will function similar to a switch.

mbroberson1 Tue, 11/10/2009 - 19:30

Hi Edison,

It's been a while since we posted on this. Have a question:

If the Metro Ethernet service the provider is provisioning is non-QinQ how much will that matter with the configuration?

Thanks

mbroberson1 Fri, 07/24/2009 - 06:53

Hi Edison,

One question comes to mind on the ME setup. With the HUB and remotes as routers and using the sub-interfaces. If you are running EIGRP over the WAN (between hub and remote sites) will any certain (special) EIGRP configuration be needed on the hub and/or remotes? Just curious.

Thanks,

Brandon

Edison Ortiz Fri, 07/24/2009 - 08:10

Brandon,

As long as you enter the corresponding subinterface subnet under the EIGRP process, the adjacency should take place without problems.

mbroberson1 Fri, 07/24/2009 - 08:21

Hi Edison,

That's what I was thinking. I just wasn't sure if for some reason I should unicast the updates to each neighbor or something else.

My reason for this was I wasn't sure if I would get the message "not on common subnet" or similar message.

Thanks

Edison Ortiz Fri, 07/24/2009 - 08:28

Each subnet has a corresponding Vlan tag. You may see this message if you mix the Vlan tag assignment on the edge routers.

mbroberson1 Fri, 07/24/2009 - 08:31

Great,

So just make sure I have the vlan tag "dot1q" numbers (for corresponding peers on same segment) assigned correctly and to the same.

Thanks,

Brandon

wilson_1234_2 Fri, 08/21/2009 - 11:09

Edison,

I have a question if we go the route you originally mentioned using switches:

The MetroE is layer 2, as I mentioned, I want to trunk a few vlans and route the others,

Would we be able to use a WAN optimization appliance on the routed vlans, which, the one we are looking at is a layer three appliance (silverpeek)?

Edison Ortiz Fri, 08/21/2009 - 11:35

I don't see why not, however I recommend consulting with Silverpeek for any caveats regarding their product and MetroE implementation.

Actions

This Discussion