We have two buildings connected through a fibre link.
One building has a L3 switch, which has been configured with VLANs for each classroom and one VLAN for the server room. We can connect between the VLANs without problem.
The second building has been configured with VLANs for each of its classrooms and another for its server room. Using router-on-a-stick, these VLANs can interconnect.
The fibre connection between the server rooms has been configured to carry all the VLANs tagged using 802.1q.
All VLANs are unique.
However, we cannot get connection between a VLAN in building 1 and another VLAN in building 2.
I suspect that this is because there is no port in the Building 1 L3 switch that is the untagged (default gateway) for VLANs in Building 2 and, similarly, the L2 switch in Building 2 has no untagged (default gateway) port for VLANs in Building 1.
Is it necessary for me to configure the router-on-a-stick in Building 2 to perform all the inter-VLAN routing? I'd prefer not to do this as most of the traffic will remain within the discreet buildings but we do require occasional traffic between the buildings.
I'd appreciate any suggestions regarding configuring inter-VLAN routing retaining the L3 switch inter-VLAN routing in Building 1 and the router-on-a-stick inter-VLAN routing in Building 2, if practical.
I am not sure that I have a full understanding of your situation. I think I understand that building 1 has a layer 3 switch doing inter vlan routing and it works ok within that building. And I think I understand that building 2 has a layer 2 switch with VLANs and a router doing router on a stick for building 2 and it works ok within that building.
I am not clear whether it is a generic problem where anyone/any VLAN in building 1 does not communicate with anyone/any VLAN in building 2 (I think this is not the case but am not sure). Or is it a single VLAN in building 1 can not communicate with a single VLAN in building 2? Or are there a couple of VLANs but not all VLANs in building 1 do not communicate with a couple of VLANs in building 2?
I am also not clear on the distribution of VLANs between the buildings. How many VLANs are present/active in both buildings, how many VLANs are present/active only in building 1 and how many are present/active only in building 2?
My guess at this point is that the issue is that there is a VLAN(s) in building 1 that is not present in building 2, and some VLAN(s) in building 2 that is not in building 1. And that neither the layer 3 switch in building 1 nor the router on a stick in building 2 have virtual interfaces or any routing information for the VLAN(s) present only in the other building.
If my guess is right you should be able to solve your problem either by configuring some routing information in both layer 3 devices (or run a dynamic routing protocol) for the missing subnets. Or you could configure virtual interfaces on one or both of the switch and router on a stick for the missing VLANs. This would allow one or both layer 3 devices to forward traffic over the trunk for the VLANs present in the other building.
Thanks for your suggestions.
We were running a single Class A network for all the classrooms and the server rooms. As the students have administrative access to the machines for the subjects that we teach, we needed to isolate each classroom from the others to limit the impact of student mistakes. I considered the solution to be to use VLANs and subnetting the Class A network to separate Class C subnets for each classroom and server room.
No VLANs are present/active in both buildings. However, the server rooms each house peer Domain Controllers for the training domain so these VLANs need to be routed. Additionally, there are occasions when the Network Admins need to route between the server room VLANs and the classroom VLANs. Generally, these can be confined to the same building but, occasionally, it may be necessary to connect to member servers in the other building.
I tried configuring some routing information in both layer 3 devices - easy on the Cisco router but not so easy on the L3 switch (Extreme Networks Summit 400-48T) because the only way to assign an IP address (for use as the next hop) to the port, connecting the buildings, is to create a VLAN for that connection.
Can you elaborate on configuring virtual interfaces for the missing VLANs, please? I am not sure what you mean or how to do what you suggest. Probably just me being a little dense but a pointer would be appreciated.
For the router on a stick you would configure subinterfaces for each VLAN in the other building that you want to access. So it might look something like this:
encapsulation dot1q n
ip address a.a.a.a m.m.m.m
This would allow anyone in the building with the router to access resources in VLAN n in the other building. To allow access from the other building to resources in the building with the router on a stick you would need a default route on the layer 3 switch pointing to an address on the router as its next hop (or as an alternative you could have the end stations in the other building configured to use the IP address of the router in that VLAN as their default gateway - it sort of depends on how you want access in the other building to work).
Thanks for the response.
I tried setting up the router-on-a-stick subinterfaces for the VLANs in Building A. This appeared to work alright except that I had used a last octet value of 254 for the untagged connections for each VLAN on the L3 switch in Building A and as the Default Gateway for the hosts attached to it. I had to use a last octet value of 253 for the subinterface on the router but didn't want to change the Default Gateway addresses on the hosts.
The L3 switch in Building A does not support subinterfaces so setting it up for the VLANs in Building B was not so straight-forward.
Having given the requirement further thought, this is what I plan to try:
1. Set the connection between Building A and Building B as a trunk connection for all VLANs in Building A and Building B (tagged for all relevant VLANs).
2. Create a VLAN for BuildingB on the L3 switch in Building A, covering a supernet for all the subnets in Building B, set the port on the L3 switch in Building A with its IP address set to the top address in the supernet and make the port untagged for that VLAN.
3. Create a VLAN for BuildingA on another subinterface on the router in Building B, covering a supernet for all the subnets in Building A with its IP address set to the top address in that supernet and make that subinterface untagged for that VLAN.
I anticipate that this configuration will allow the VMware ESX Servers in each building be able to work with the VLANs in either building - by having the connection configured as a trunk connection - all VLANs tagged.
I anticipate that it will allow the L3 switch in Building A to route between the Building A VLANs, and between them and the supernetted BuildingB VLAN. Any traffic on the BuildingB VLAN will leave Building A via the Buildings' interconnection port, which will untag it and pass it through the interconnection to the network in Building B where it will get tagged as it leaves the connecting port with the VLAN tag for the BuildingA VLAN, pass it via the other trunk port to the router-on-a-stick, which will route it to the relevant Building B VLAN.
Similarly, I anticipate that the router-on-a-stick subinterfaces will route between the Building B VLANs, and it will route between those VLANs and the supernetted BuildingA VLAN. Any traffic for that VLAN will be passed to the Buildings' interconnection port in Building B, have its tag removed, pass it to the interconnection port in Building A, which will route it to the relevant Building A VLAN, tagging it with that VLAN tag.
I hope that this makes sense. Can you see any problem with it, please?
I shall update this forum message when I have tested the configuration.
Thanks again for your assistance.
I have read this post several times and am still having some difficulty understanding what you intend to do. So let me address a couple of things and make a slightly different suggestion.
- at one point you describe the fiber link between buildings as being a trunk with dot1Q tagging. Then as you describe how packets will flow you describe them as being untagged for transmission over the fiber and then re-tagged.
- I am not clear how your idea of a supernet VLAN will work and suspect that there are problems with it. On the router on a stick if there is a VLAN that consists of a supernet of the addresses in the other building then the router will believe that all of those addresses are local and that it should be able to ARP for any of those addresses and reach them as a local address. I do not believe that this is the case.
I have been thinking about your idea of a supernet and believe that there might be a slightly different way to use this. If it is the case that all of the VLANs and their subnets in building A can be grouped in a unique supernet and that all of the VLANs and their subnets in building B can be grouped in a unique supernet, then it should be possible to configure the fiber link not as a trunk but as a routed link or as a single VLAN where the layer 3 switch and the router both have addresses and can talk to each other. They will each know how to access their own subnets. Then you should be able to put a static route for the other building supernet and use the other device address as the next hop for the static route.
Thanks again for your response.
Unfortunately, I only had one hour of trial time this week and next week looks to be the same.
I understand what you mean in your second paragraph and I'm certain that it would permit routing between the two buildings. However, it won't meet all the requirements.
We have two ESX servers, one in each building, which are to provide centralised delivery of VMs to the classrooms in their respective building via the classroom VLAN (the virtual switch on the ESX server can be configured for a specific VLAN before the VM is started). Occasionally, there may be a need for the ESX server in one building to provide centralised delivery of VMs to other building (e.g. if the other building's ESX server is down for any reason).
For this reason, I have to configure each ESX Server to be aware of all of the VLANs (for its own building classrooms and for the classrooms of the other building), and I need to trunk the classroom VLANs over the fibre connection between the buildings.
In addition, each building has their own Domain Controller for the domain that provides general services to the joint classroom environment and these need to perform their AD synchronisation although they are in their own separate Server Room VLANs. The Classroom VLANs also need occasional access to the servers in that domain. For this reason, I need to provide routing between all VLANs although, most of the time, the routing will be localised to the specific building.
A single /8 network has been configured such that each building uses a different value for the second octet and each classroom uses a different value for the third octet - each classroom being a /24 subnet of the Class A network and has its own VLAN. This arrangement enables me to consider using a different /14 supernet for each building.
This was the principle behind my thought of creating a VLAN in BuildingA for the supernet for BuildingB and configuring the port used for inter-building connection on the L3 device in BuildingA to use the highest available IP address in the supernet for routing purposes and to untag the traffic. Similarly, creating a VLAN in BuildingB for the supernet for BuildingA and configuring the interconnecting port in BuildingB to untag the traffic for that VLAN and configuring a subinterface on the router-on-a-stick in BuildingB with the highest address in that supernet for routing purposes.
Non-routed traffic would pass both ways retaining its original tagging over the trunked connection. Traffic from VLANs in BuildingA, which required routing to VLANs in BuildingB, would be routed in the L3 device (based on the IP Destination Address) to the supernet VLAN for BuildingB and the interconnecting port on the L3 device would untag the traffic. Once over the fibre connection, as the traffic left the interconnection port on the BuildingB switch, it would be tagged with the BuildingA supernet VLAN tag, pass to the router-on-a-stick and be routed to the respective BuildingB VLAN.
When I tried this in the one hour trial time, it didn't work but, as I've written this explanation, I think that I've realised that I misconfigured the IP setting for the subinterface for the supernet VLAN. I'll check that next Friday when I have my next one hour trial slot.
You indicated that you thought that there may be problems with my idea of a supernet VLAN. Now that I've tried to describe my idea, please advise of you think that the problems that you could see persist or if there is a different problem that may have caused the failure to route between BuildingA's VLANs and BuildingB's and vice versa other than my potential misconfiguration of the IP settings.
As I think about it I continue to believe that there are problems with your theory of creating a VLAN for the supernet of the other building. I believe that one of the problems is that if you create a VLAN in building A for the supernet of building B then the layer 3 device will believe that the addresses in the supernet VLAN are local and that it should be able to ARP for them. But I do not believe that it will be successful in its effort to ARP for them. And the failed ARP will cause it to drop the packet.
Also I believe that there is another problem. If you create a VLAN in building A for the supernet of building B, will that VLAN exist in building B? (I believe that it must). But then you have a VLAN with an /16 address for the collection of subnets but you also are trying to configure the individual VLANs for classrooms etc with their /24 subnets. But you can not configure 2 interfaces with overlapping addresses. Perhaps an example will help: lets assume that building A is supernet 10.1.x.x and that building B is supernet 10.2.x.x. So there will be a VLAN for the supernet with router address 10.2.0.254/16 (or something like that). But when you attempt to create a VLAN for a classroom with address 10.2.1.254/24 you will get an error message about overlapping addresses.
Thanks again for a helpful response.
I think that my idea may be betraying a fundamental misunderstanding that I have. I thought that, if I had a VLAN in BuildingA for the supernet in BuildingB (tag 421), this would enable the L3 device to route any traffic from a Classroom VLAN in BuildingA to the BuildingB VLAN. When that traffic appears at the port that interconnects with BuildingB, as that port was configured for the VLAN untagged, it would untag the traffic, removing the dot1q header and resorting to the Ethernet packet headers. These would contain the SA from the L3 port that acted as the gateway for the classroom VLAN in BuildingA and have a DA of the device to which it wished to communicate in BuildingB. When it arrived at the port in BuildingB at the other end of the interconnection, as it had no dot1q header but did have a SA from BuildingA, the port would tag it with the dot1q header for the BuildingB VLAN for the BuildingA supernet (tag 662). The only other port that has any 'knowledge' of VLAN 662 is the router-on-a-stick subinterface that provides the routing from VLAN 662 to the classroom VLANs in BuildingB, so it would route the traffic to the relevant BuildingB classroom VLAN. This traffic would transfer through its r-o-a-s subinterface, getting tagged with the dot1q header for the relevant BuildingB classroom VLAN, back to the BuildingB Ethernet switch where it would arrive at the port connecting to the classroom's hub/switch uplink, get untagged because all the devices connected via that port are part of that BuildingB classroom VLAN.
Building on your example; if BuildingA uses 10.1.0.0/14 (it could just as easily be /16) and a VLAN (662) exists for this supernet in BuildingB; and BuildingB uses 10.2.0.0/14 and a VLAN (421) exists for this supernet in BuildingA. BuildingA has classroom 53 (VLAN tag 153) and BuildingB has classroom 16 (VLAN tag 116). If device 12 in classroom 53 (10.1.53.12) sends traffic to device 10 in classroom 16 (10.2.16.10), as it leaves the uplink from the classroom hub/switch through its port on the L3 device, it would normally get tagged with the dot1q header 153) but, as the relevant L3 port is also the default gateway for the devices in classroom 53, it routes the traffic to another VLAN, in this case VLAN 421 because the DA is in the 10.2.0.0/14 address space. The only port on the L3 device that 'knows' about VLAN 421 is the interconnecting port with BuildingB and it untags the traffic. At the other end of the fibre connection, the port in BuildingB tags the traffic with the dot1q header for VLAN 662 because the SA is in the 10.1.0.0/14 address space. The traffic passes through the r.o.a.s. subinterface for VLAN 662 and gets routed to the subinterface for VLAN 116. The traffic now has a dot1q header for 116, gets untagged as it passes through the port that connects to the uplink from the hub/switch in classroom 16.
I'm guessing that my perception of what happens is flawed so your assistance in redressing this misunderstanding will be much appreciated.
I've just recognised one flaw in my understanding - I was confusing L3 addressing with L2. The dot1q header gets appended before the Ethernet header (MAC addresses rather than IP addresses) - I should have paid more attention to the second part of your last response.
Back to the drawing board for me while I attempt to figure out how to allow trunking of all VLANs across the building interconnection (to permit VMs on the ESX Server in BuildingA to work with clients in a BuildingB classroom when the need exists) while providing routing between classroom VLANs in BuildingA with classroom VLANs in BuildingB. I shall re-read your earlier responses to see how they can help me to come up with a solution for both requirements.
I am glad that you are getting a better understanding of the implications of layer 2 operations and layer 3 operations. While it may be helpful to trunk all VLANs between the buildings I am not yet convinced that trunking all the VLANs is necessary.
I am sure that there are aspects of your situation that I do not yet understand. But in focusing on the requirement that an ESX server in one building (and in one VLAN) may need to communicate with clients in the other building (and in a different VLAN) it seems that the essential part of the solution is routing. To get from 1 VLAN into some other VLAN there must be a layer 3 forwarding device (router or layer 3 switch). So the solution may involve deciding how the layer 3 switch will route to the VLANs in the other building, and how the router on a stick will route to the VLANs in the other building. As long as the layer 3 switch and the router connect to each other with at least 1 VLAN in common then they should be able to do the routing. Rather than a VLAN for the supernet I suggest a static route for the supernet. If the layer 2 switch has a static route for the supernet of the other building with the router as next hop, and if the router has a static route for the supternet of the other building with the layer 3 switch as the next hop, then it seems to me that you have a workable routing solution.
Thanks once more for your interest and assistance.
Regrettably, I appear to have mislead you with regard to the ESX servers. Generally, the ESX server in BuildingA will provide centralised VMs for the classrooms in that building. Similarly, the ESX server in BuildingB will, generally, provide centralised VMs for the classrooms in that building. However, there may be occasions when it is necessary for the ESX server in BuildingA to provide centralised VMs for the classrooms in BuildingB and vice versa.
When that is required (e.g. for the BuildingA ESX server to provide centralised VMs for classrooms in BuildingB), then the relevant VMs on the BuildingA ESX server will be configured so that their virtual NICs will be part of the relevant VLAN for the classroom in BuildingB. This is why VLAN trunking is required for all VLANs. The only time that BuildingB VLANs will appear on any ports in BuildingA will be at the Virtual Switch on the BuildingA ESX server, trunked through the L3 switch ports that connect to the BuildingA ESX server. No other port on the BuildingA L3 switch will 'know' about BuildingB classroom VLANs.
When I get my hour on Friday, I'll concentrate on getting routing working across the connection between the buildings using your suggestion about using 1 VLAN in common - I'll try with a single VLAN, which is separate from the other VLANs, e.g. only a single VLAN across the connection rather than separate VLANs (421 and 662) either way. Once I've achieved that, I'll see what can be done to run BuildingB classroom VLANs on the ESX server in BuildingA and vice versa because this should be a less common occurrence.
Thanks again and I'll update this once I've had my one hour on Friday unless you make any other suggestions meantime.
It is clear that I am still not understanding your environment and your requirements. At one point I thought that each VLAN was contained within a building and that you needed connectivity from a device in a VLAN in one building to some device in a different VLAN in the other building. For that situation I believe that a routing solution would provide what you need. From this recent post I believe that I now understand that the situation is quite different. If the ESX server in building A will configure its virtual NIC to be a member of a VLAN in building B, then the VLAN from building B must be present on the switch in building A, and the connection from the switch to the ESX server must be a trunk that carries every VLAN for both buildings.
So if my understanding is correct the solution will include at least these elements: the switch(s) in each building must be aware of and carry every VLAN (from both buildings), the link between buildings must be a trunk that carries every VLAN, the connection from the building switch to the ESX server would be a trunk that carries every VLAN.
Thanks for your latest update.
There are four separate requirements:
1. There are separate Domain Controllers in each building for the primary domain, each in its own VLAN for its serverroom. These DCs need to keep AD in sync. and, therefore, a need exists to route between these separate serverroom VLANs.
2. There are occasions when the hosts in the classroom VLANs need to interact with servers in the primary domain. There is, therefore, the need for the classroom VLANs to be routed to the primary domain's VLANs. Most of the time, these are limited to their specific building but there may be occasions when it is necessary for the servers in the other building to handle these requests.
3. Most of the time, the separate classroom VLANs will be limited to their specific building with the ESX server in each building providing centralised VM environments for the classroom VLANs in that building.
4. Occasionally, there may be a need for one building's ESX server to provide a centralised VM environment for clients in a classroom VLAN in the other building.
As I see it, this requires the following solutions:
1. A routing solution between the serverroom VLANs in each building.
2. A routing solution in each building so that its classroom VLANs can route to that building's primary VLAN and, then, route across the routing solution above to the serverroom VLAN in the other building.
3. This solution exists because all the VLANs in each building are self-contained within it - we have that working currently.
4. A trunking solution to carry one building's VLAN (tagged) traffic across to the other building; once there, get passed via other trunk ports to its ESX server and use its virtual switch to connect to the centralised VM environment, which is running in the VLAN for the other building's classroom.
I hope that this description clarifies the requirements that I am facing.
Continuation of previous post.
During the one hour that I had this Friday, I tried unsuccessfuly to create a routing solution between the buildings. I had time to try two different approaches:
1. I continued to use the L2 switch port in BuildingB as one end of the interconnection. I created a single VLAN for the interconnection, called "connect" and using tag 500. I allocated 10.3.0.1 255.255.255.252 for the port in BuildingA and 10.3.0.2 255.255.255.252 on a separate subinterface f 0/0.500 in BuildingB, which was configured for encap dot1q 500. I also configured a static route using 10.1.0.0 255.255.0.0 10.3.0.1 in BuildingB but could find no way to set a static route in the L3 device in BuildingA - more manual checking required. Within each building, the other subnets could ping their respective 10.3.0.0 port address but could not ping the remote port address.
2. I changed the interconnecting cable from the L2 switch to a spare Fast Ethernet port on the r-o-a-s in BuildingB. I removed the subinterface for the 500 VLAN on int f 0/0.500 and created int f 0/1.500, encap dot1q 500, configuring its IP 10.3.0.2 255.255.255.252 and setting a static route 10.1.0.0 255.255.0.0 10.3.0.1. Same results when attempting to prove the solution using ping.
The problem with attempt one is that the L2 switch port was still acting as a trunk port for all the VLANs in BuildingA and BuildingB, as was the port on the L3 device in BuildingA.
In attempt two, there were no trunked VLANs on the connection in BuildingB but I used a subinterface on Fast Ethernet 0/1 because that is the only way that I can configure dot1q encapsulation for the "connect" tag 500 VLAN.
With time running out, I restored the original configuration.
I'm considering using the spare Fast Ethernet interface on the r-o-a-s again but not using a VLAN for the interconnection, setting the IP configuration on the f 0/1 interface and setting the static route. I'll discover how to set an IP address on the port on the L3 device and how to set a static route on it. If that works, I'll create subinterfaces on the f 0/1 interface for each of the VLANs in BuildingA to see whether I can trunk those VLANs between the buildings. I'll take it from there if that works. If not, it'll be back to the drawing board.
Any other suggestions will be gratefully received.
Here is the latest status on my attempts to achieve the four requirements that we have.
I've managed to get the L3 device in BuildingA to communicate with the R-O-A-S f 0/1.500 sub-interface, as described above. I don't know why it didn't work the last time - obviously, finger-trouble on my part.
So, from my first post on Apr 19, I've now got solutions 1-3 working.
I'm puzzling over solution 4, however, because the only connection between the buildings now is a routed connection between f 0/1.500 on the R-O-A-S and the other Fast Ethernet interface on that machine. (The fibre connection between the buildings is a Layer 2 Ethernet connection from the port in the L3 device and the f 0/1 interface on the R-O-A-S). I'm not sure that I'll be able to trunk all the VLANs over a routed connection, seeing that VLAN trunking occurs at Layer 2 and, of course, routing occurs at Layer 3.
Is it possible to trunk VLANs over a routed interface?
If anyone has done this, please describe how.