Windows NLB and Cisco to Nortel ether trunk

Andy White · ‎03-03-2010

Hello,

I'm having a issue creating a Windows NLB config (Network Load Balance). I have 2 3750 switches, each 3750 trunk (ether channel) into a Nortel 8600 core switch. I have created a new VLAN on the Nortel's and on the Cisco 3750's and I can ping across from a PC in the Nortel to a PC in the Cisco's.

Now I have installed 2 Windows 2008 new servers, one server plugs into one Cisco switch and the other, but they both go into the same VLAN. Ihave created a multicast NLB and I can see mac and IP in the arp table:

Switch A

Internet 192.168.19.3          154   xxx.xxx.1303 ARPA   Vlan1019
Internet 192.168.19.2           75   xxx.xxx.23c0 ARPA   Vlan1019
Internet 192.168.19.1           78   xxx.xxx.2339 ARPA   Vlan1019

SwitchB

Internet 192.168.19.3          155   xxxx.xxxx.1303 ARPA   Vlan1019
Internet 192.168.19.2           76   xxxx.xxxx.23c0 ARPA   Vlan1019
Internet 192.168.19.1           79   xxxx.xxxx.2339 ARPA   Vlan1019

192.168.19.3 is the virtual NLB address

Switch A & B:

Can't ping 192.168.19.3 but can the 2 physical IP's (19.1 and 19.2)

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.19.3, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Switch#ping 192.168.19.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.19.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/8 ms
Switch#ping 192.168.19.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.19.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/8 ms

On the Nortels they can ping all the IP's, but no users plugged into the Nortels can.

Nortel Core 1:

192.168.19.3    xxx.xxx.1303 1019     -    DYNAMIC 1241
192.168.19.2    xxx.xxx.23c0 1019 MLT 12 DYNAMIC 2158
192.168.19.1    xxx.xxx.2339 1019 MLT 11 DYNAMIC 2158

Nortel Core 2:

192.168.19.3    xxx.xxx.1303 1019     -    DYNAMIC 1635
192.168.19.2    xxx.xxx.23c0 1019 MLT 12 DYNAMIC 1653
192.168.19.1    xxx.xxx.2339 1019 MLT 11 DYNAMIC 1636

Both cores can ping the physical IP's and virtual

CORE# ping 192.168.19.3
192.168.19.3 is alive
CORE# ping 192.168.19.2
192.168.19.2 is alive
CORE# ping 192.168.19.1
192.168.19.1 is alive

My PC in the Core's can't ping any of these IP's. The weirdest thing of all is we already have a NLB setup on 2 other servers in a different VLAN in the Cisco switches and it all works and I have (I think) set it up the same. I can only think something static (mac) was entered somewhere on the Nortel Cores as I can't see anything on the Cisco's. I think the NLB is broadcasting ok other the ether channel as the Nortel cores are picking up the arps.

Maybe I should post on the Nortel forums as I suspect it's that.

Giuseppe Larosa · ‎03-03-2010

Hello Andy,

you have hidden the only useful information in this scenario:

to what multicast address have you mapped the MS NLB VIP 192.168.13 ?

you need to use a multicast MAC address that is not within the range of MAC adddresses used for multicast IPv4 addresses or IGMP snooping will create problems

a MAC is multicast if less significant bit of most significant byte is set to 1 like in 01xx.yyzz.kkdd

then you need a static mapping to the ports in direction of real servers part of the cluster

see

http://www.cisco.com/en/US/partner/docs/ios/ipv6/configuration/guide/ip6-addrg_bsc_con_ps6441_TSD_Products_Configuration_Guide_Chapter.html#wp1334130

>>Ensure that you use the multicast mode on the NLB cluster. Cisco recommends that you do not use multicast MAC addresses that begin with 01 because they are known to have a conflict with the IGMP setup.

Edit:

right link is :

http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml#mm

Hope to help

Giuseppe

Andy White · ‎03-03-2010

Hi,

Thanks for spending the time to reply!

I did mention the VIP "192.168.19.3 is the virtual NLB address"

192.168.19.1 is once server

192.168.19.2 is the other server

1.) I am using multicast. Are you saying I need to use a MAC multicast address for the VIP that is not similar to the MAC address for the physical NICs used in the NLB cluster? (192.168.19.1 and 192.168.19.2)? If so how do I change the VIP MAC address?

2.) Can I give you the real MAC address to help?

3.) What do I need to statically map on the 2 Cisco 3750's?

Sorry for all the questions, I've been pulling my hair out and now have hope

Giuseppe Larosa · ‎03-03-2010

Hello Andy,

1) I'm saying the multicast address must be different from those used to transport IPv4 MAC addresses, you can derive it from NIC MAC address using for example 03 as first byte

2) the document I've linked explains what to do:

something like:

mac-address-table static 0300.5e11.1111 vlan 200 interface fa2/3 fa2/4 disable-snooping

then a static ARP entry completes the solution

arp 172.16.63.241 0300.5e11.1111

adapt the command above to your scenario including access ports and trunk ports to other switches

Hope to help
Giuseppe

Andy White · ‎03-03-2010

That link didn't work for me once I logged in.

Are you basically saying 0300.5e11.1111 vlan 200 interface is on ports fa2/3 fa2/4 and has an IP address of 172.16.63.241?

How would this look for my scenario on both switches:

multicast mac 0300.3333.3333

Multicast VIP 192.168.19.3

server nic mac 0300.2222.2222

IP 192.168.19.2

server nic mac 0300.1111.1111

IP 192.168.19.1

ether tunk on both switches are:

switch 1 - fa12 & fa24

switch 2 - fa12 & fa24

Giuseppe Larosa · ‎03-03-2010

Hello Andy,

sorry I usually edit links to allow general access, this time I've missed amd I've posted also a wrong link.

Have a look at:

http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml#mm

Hope to help

Giuseppe

Kevin Dorrell · ‎03-04-2010

One thing I found with NLB=multicast+IGMP was that I had to configure static ARP entries in the router for the NLB virtual address. When the router did an ARP for the virtual address, the server(s) responded with their multicast MAC, but the router did not believe it, and did not populate its ARP table. So the virtual IP was not accessible outside its own VLAN. Static ARP entries fixed that.

On the other hand, strangely, my multicast MACs started with the more conventional 01:00:5e, and not 03:00:5e. In fact, the MAC addresses were 01:00:5e:7f:xx:yy, where xx:yy are the hex representation of the last two octets of the NLB virtual IP address.

Kevin Dorrell

Luxemborg

Andy White · ‎03-04-2010

Hi,

What is strange on my setup, I already have a NLB (multicast) set up, I'm trying to do the exact same thing. If I look on both Cisco switches the only static mac entries are these:

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
All    0100.0ccc.cccc    STATIC      CPU
All    0100.0ccc.cccd    STATIC      CPU
All    0180.c200.0000    STATIC      CPU
All    0180.c200.0001    STATIC      CPU
All    0180.c200.0002    STATIC      CPU
All    0180.c200.0003    STATIC      CPU
All    0180.c200.0004    STATIC      CPU
All    0180.c200.0005    STATIC      CPU
All    0180.c200.0006    STATIC      CPU
All    0180.c200.0007    STATIC      CPU
All    0180.c200.0008    STATIC      CPU
All    0180.c200.0009    STATIC      CPU
All    0180.c200.000a    STATIC      CPU
All    0180.c200.000b    STATIC      CPU
All    0180.c200.000c    STATIC      CPU
All    0180.c200.000d    STATIC      CPU
All    0180.c200.000e    STATIC      CPU
All    0180.c200.000f    STATIC      CPU
All    0180.c200.0010    STATIC      CPU
All    ffff.ffff.ffff    STATIC      CPU

from the Cisco switches I can ping the 2 physical IP but not the virtual NLB, but from the desktops I can ping all which are in the Nortels. From the Nortels I can ping all. I winder if the Nortel consultants add something static to their configs.

If I need to add static arp entries, do I just add the all 3 mac address (2 physical and 1 virtual MAC) to the single port where the server is plugged into on both switches?

Kevin Dorrell · ‎03-04-2010

You are right, a PC works OK because it believes the MAC address in the ARP response, and maybe the Nortel does the same. It is only Cisco that does not believe a MAC multicast in an ARP response.

The static ARP entries need to go on whatever Cisco device is handling the routing into the VLAN that is hosting the NLB server. You don't need static entries for the individual nodes because they will respond to ARP with their own built-in (unicast) addresses. It is only the virtual address that needs to be statically mapped because it is the virtual address that uses multicast.

The MAC address is usually 01:00:5e:7f:xx:yy, where xx and yy are the last two parts of the virtual IP address, expressed in hex. If in doubt, put a PC on the same VLAN as the NBL, and ping for the virtual IP, then look at the PC's ARP cache.

BTW, are these switches layer-2, or are they handling the routing? If they are layer-2 only, then their ARP caches are not relevant. It is the ARP cache on the routing engine you need to be looking at.

Also, I would comment that the table you have shown is the MAC forwarding table, which is not the same as the ARP table. For the ARP table, go to the router (or layer-3 switch) and do show arp. The ARP table shows the mapping between IP address and MAC address.

Kevin Dorrell

Luxembourg

Andy White · ‎03-04-2010

I still can't ing fromt he Nortel, but I add the following and I can ping the NLB from the switches now:

mac address-table static 03bf.c0a8.1303 vlan 1019 interface gigabitEthernet 2/0/21

arp 192.168.19.3 03bf.c0a8.1303 arpa

What is strange the NLB IP (192.168.19.3) shows no VLAN assign to it like 1019

Internet 192.168.19.3            -   03bf.c0a8.1303 ARPA
Internet 192.168.19.2            0   0022.1964.23c0 ARPA   Vlan1019
Internet 192.168.19.1            0   0022.1964.2339 ARPA   Vlan1019