Inter-vlan routing and gateway issues

[bavingtonm]

Hi

Please please advise!!

I have a problem with inter-vlan routing and using the correct gateway.

I have a network configured with 8 Vlans, each vlan is in a different subnet and resides in separate buildings in the campus.

Example.

VLAN 10 - 10.217.6.0/24 - ICT BLOCK

VLAN 20 - 10.217.7.0/24 - N-BLOCK

VLAN 30 - 10.217.8.0/24 - LIFT BLOCK

VLAN 40 - 10.217.9.0/24 - ADMIN BLOCK

VLAN 50 - 10.217.13.0/24 -TEC-BLOCK

VLAN 60 - AIRSPACE NETWORK

VLAN 70 -N 10.217.12.0/24 - SERVERS

VLAN 100 - 10.217.20.0/24 - MANAGEMENT VLAN

The network core comprises of 2 x 3750G-12S switches trunked together via 2 x fibre links and is configured as a collapsed core/distribution.

Each vlan has a trunk configured on each of the cores and also SVI interfaces for each vlan.

I have created 1 virtual address for both the core switches on the 10.217.12.0/24 network, where Core_sw_1 has an address of 10.217.12.21/24 and Core_sw_2 has an address of 10.217.12.22/24, they now share a virtual address of 10.217.12.50/24.

This 10.217.12.50 address is being used as a default gateway for all the vlans and servers. With this gateway in place and the use of a few IP Helper-address commands, every vlan/subnet can communicate with each other, log into the servers on the 10.217.12.0/24 network and also access the internet perfectly.

But… 4 of the servers (Webmail, Webserver, etc) require inbound access from outside the LAN. But if I have these servers configured with the gateway of 10.217.12.50/24 then nothing can connect or access from the outside, but the LAN works good. If I change these servers DG to 10.217.12.1/24 (which is the address of the companies ISP owned/managed router) then remote and webserver access works perfectly but every other vlan on the LAN can no longer log into or even ping these servers, with the exception of the 10.217.12.0/24 network (vlan 70).

Question:

How can I get all vlans/subnets to communicate with the servers when the servers are using a DG of 10.217.12.1/24 and everything else uses a DG of 10.217.12.50/24?

Please help as it is causing a lot of downtime and not to mention frustration!

Thank you.

Mark

[JORGE RODRIGUEZ]

Mark,

We need to break down the logical design to understand what could be the problem. I believe the issue lies in the way you have configured each SVI , virtual IPs for each SVI VLAN in relation to dual core, as well as how is your default route configured on the core switches. If you could provide a topology diagram that would also help.

You said the core is composed of two 3750G-12S switches with a trunk link between the two, so far this seems clear. It is not so clear when you said Each vlan has a trunk configured on each of the cores and also SVI interfaces for each vlan. perhaps you meant each access switch has a trunk link configured to each core , if this is so , so far is clear as this seems standard dual core design.

I believe you have one of your problems here in the description of one of your paragraph on the design of each vlan, which is also causing internal routing problems.

This 10.217.12.50 address is being used as a default gateway for all the vlans and servers. With this gateway in place and the use of a few IP Helper-address commands, every vlan/subnet can communicate with each other, log into the servers on the 10.217.12.0/24 network and also access the internet perfectly.

You want to configure each vlan with a virtual address in the case of hsrp for providing core switch failover architecture. Each SVI in your core switches will have virtual IP in which it will be provided to hosts as a default gateway, that is.

If you have in the case of VLAN 10 - 10.217.6.0/24 -

configuration with hsrp would be something as:

example.

CORE_switch_3750G_1

interface vlan10

ip address 10.217.6.2 255.255.255.0

standby 10 ip 10.217.6.1

standby 10 preempt

standby 10 priority 110

standby 10 timers 5 15

CORE_swtich_3750G_2

interface vlan10

ip address 10.217.6.3 255.255.255.0

standby 10 ip 10.217.6.1

standby 10 preempt

standby 10 timers 5 15

Hosts under this 10.217.6.0/24 subnet will use default gateway of 10.217.6.1 , the same would be done for each of the SVI's you have create. For the remaining VLANs in your core switches using above example.

For VLAN 20-10.217.7.0/24 - N-BLOCK 10.217.7.1 would be default gateway for Hosts.

For VLAN 30-10.217.8.0/24 - LIFT BLOCK 10.217.8.1 would be default gateway for Hosts etc..

HSRP support page:

http://www.cisco.com/en/US/tech/tk648/tk362/tk321/tsd_technology_support_sub-protocol_home.html

Now from the CORE switches where is your internet edge router is there a firewall upstream from the core switches. Your core switches will need to be configured with a default route pointing to your next upstream device.. assuming you have a router as an up stream and assuming you are doing static routing you would configure a default route in your core switches :

Ip route 0.0.0.0 0.0.0.0

I could suspect that If your upstream internet router IP address happens to be under 10.217.12.0/24 then your default route for the core switches would be 10.217.12.1

Again, if you could provide a visual topology or description it would help us to decipher the issues.

Other question is how are the outside connecting to the inside, are you NATing via the edge internet router? Is there a firewall conducting NATing for inbound connections from the internet?

Rgds

Jorge

[bavingtonm]

Hi

Thanks for the detailed reply.

I have attached a very rough topology in word format. The outside edge router is manageed by an ISP and is an Extreme networks switch. The details of the config are a mystery, no one on site can supply me with this info.

I do however know that NAT takes place on this Extreme switch and it has 3 ports configured. 1 port connects to a LES 100 NTU, 2 is configured with an address of 10.217.12.15 (which is used for the proxy server to go out on)and the 3rd is the 10.217.12.1 which I believe is for outside traffic as its unfiltered. Incedently all hosts on each vlan go out to the internet via the proxy server. I have been told that the ISP takes care of the firewall function at their end, but how true this is, I'm not sure!

I have configured HSRP but making the 12.50 the default gateway seems a mistake on my part. I was originally using the address of the SVI configured on the cores, and not the HSRP virtual router address as you suggested. For example i was using 10.217.7.1 (which is the SVI on Core 1) for the gateway on vlan 20 (10.217.7.0/24 network) but should of been using 10.217.7.5 which is the HSRP router address, if I understand you correctly?

On the Cores I have configured:

IP Route 0.0.0.0 0.0.0.0 10.217.12.15

I think its probably a good idea to attached a few sh runs too. So I have attached the sh runs for both cores and a few sh runs of different vlan switches. Incedently on the top switch in the cabinet for each vlan there is an IOS DHCP server configured for that subnet, all the other switches below do not.

Please keep the suggestions coming!!

One last question, If I use each Vlans HSRP router address as the gateway for the subnet, will I need to add any static routes?

Many thanks

Mark

[JORGE RODRIGUEZ]

Mark thanks for the additional information, definitely helps !

At a glance hsrp in SVI configuration seems fine. There is nothing wrong using any IP for the virtual IP as long it is consistant on the HSRP configuration and used on hosts as their default gateway, Virtual IP could be any IP within the chosen subnet as long is not used by any other hosts.

ON the core switches you have a default route pointing to proxy server 12.15. Unfortunately I have not worked with proxy setups and I would guess that there should be a way to bypass proxy for the server VLANS but again I have not played with Proxy to better suggest perhaps someone here could comment on it. Here the proxy could be the issue for the traffic from outside to inside have to hit the proxy but I could be wrong.

I would however suggest that for server vlans perhaps bypass the proxy 10.217.12.15 and use the 12.1 as default route for vlan70 ,you could use route-map using PBR and apply the route-map to server vlan to have 10.217.12.1 as the default route, this is just a thought for bypass proxy.

Server vlan VLAN 70 -N 10.217.12.0/24 - SERVERS

Perhaps something as:

CORE_1

route-map vlan70 _noproxy permit 10

match ip address 10

set ip default next-hop 10.217.12.1

interface vlan 70

ip policy route-map vlan70 _noproxy

access-list 10 permit 10.217.12.0 0.0.0.255

CORE_2

route-map vlan70 _noproxy permit 10

match ip address 10

set ip default next-hop 10.217.12.1

interface vlan 70

ip policy route-map vlan70 _noproxy

access-list 10 permit 10.217.12.0 0.0.0.255

or more granular acl using per server Ip

access-list 10 permit 10.217.12.100 0.0.0.255

access-list 10 permit 10.217.12.101 0.0.0.255

etc..

pbr reference

http://www.cisco.com/en/US/docs/switches/metro/catalyst3750m/software/release/12.2_44_se/configuration/guide/swiprout.html#wp1210866

One last question, If I use each Vlans HSRP router address as the gateway for the subnet, will I need to add any static routes?

You don't need additional static routes as there is already defined gateway of last resort being the 12.15.

On the other hand I would also contact the ISP that manages the ISP extream network router as they are the ones who will provide you with accurate info about NATing for inbound hosts as well as the proxy setup.

Rgds

Jorge

[bavingtonm]

Hi

Thanks again, this is very helpful.

I thought I mentioned that there is an ethernet cable attached from the router using address 10.217.12.1 that connects directly to the server switch. This bypasses the proxy server. The LAN side proxy address is 10.217.12.15 and the interface that links to the router uses an address of 10.217.10.1 (router) and 10.217.10.10 (proxy).

Ok, so in your suggestion of using route-map using PBR, well where would I configure this command on core 1?

route-map vlan70 _noproxy permit 10

match ip address 10

set ip default next-hop 10.217.12.1

Would this be configured under global config mode?

I peresume this command is entered in Int Vlan 70 conf mode?

ip policy route-map vlan70 _noproxy

and the access lists in global config mode?

Many thanks

Mark

[bavingtonm]

Sorry for the barage of questions, but one last thing.

With the route-map commands configured on vlan 70, what gateway should I give the servers NIC now?

Many many thanks for all your help with this matter, its given me a new angle to try!

Mark

[JORGE RODRIGUEZ]

Sorry Im back.. bear with me!!

the acl and route maps are done in global configuration mode , as soon as you enter 1st statment you will get the pbr route-map prompt, then apply the route map to the SVI interface.

The servers default gateway should always be the HSRP IP of the SVI interface.

route-map vlan70 _noproxy permit 10

match ip address 10

set ip default next-hop 10.217.12.1

I would suggest to create a test SVI for testing PBR place a laptop or something in test vlan.. it is also important to get information from the managed extreme router for testing the NATing inbound towards the test system.

Rgds

Jorge

[bavingtonm]

Hi Jorge

Ok, I have attached another diagram of how the links to the router and proxy connect to the switch on vlan 70, it mght helpfor my next question.

If I keep the servers gateway as 10.217.12.50 (the HSRP router address)and then add pbr route-maps to go out via 10.217.12.1, then how can I ensure outside connections still work, when they seem to require the servers gateway to be 10.217.12.1?

Sorry for yet another question, but I must be clear on this!

Many thanks

Mark

[JORGE RODRIGUEZ]

It should work, the diagram connections indicates proxy server has two interfaces one on 10.217.12.15/24 network which is your default route for core switches and another interface out to extreame switch router on the 10.217.10.0/24 network. The core switch has another connection on the 10.217.12.0/24 network going to extream switch router 10.217.12.1 interface as another gateway.

the pbr will simply force vlan70 hosts servers to use 10.217.12.1 as their default route for outbound internet thus not using 12.15 proxy gateway which will be handle by extream switch-router while maintaining the rest of vlans go through proxy server at 12.15 . A way to test like I said would be to create a test SVI vlan to test it or even use a single host in your acl on the vlan 70 for testing outbound connection using prb.

And of course that 10.217.12.0/24 is properly NATed in extream or firewall for internet outboung connections.

Rgds

Jorge

[bavingtonm]

Hi

I have started to configure the route-maps on the cores but when I add the

command "ip policy route-map vlan70 _noproxy" under int vlan 70 config mode, the switch will not allow the command. It dispays an "^" under the "O" in the work Policy!

I have looked to see the supported commands by typing "IP ?" and the word "Policy" isn't listed, what should I type now?

Just to let you know the core switches are using 12.2.(35)SE5 IPBase IOS, any more ideas please?

Many thanks

Mark