Routing - odd behaviour

[#1]

I'm hoping somebody can point me in the right direction for a problem I'm experiencing... I have attached a basic diagram to hopefully make sense of my explanation of the problem.

I have 3 3560 switches which are configured with trunks between them. They run vlan 10, 11 & 12.

I have a 'core' switch (switch 1) of these 3 to which an MPLS router is connected on vlan12.

I in addition have another switch hanging off the 'core' switch via a routed link (switch 4). I have EIGRP configured as a stub and as such the IP address on the routed link at the core switch end is of a /24 from vlan 1 on the other switch. This makes the route directly connected and therefore distributed via EIGRP stubs.

Switch 1 is then exchanging routes with the MPLS router (via EIGRP).

The problem I have is that from any subnet on any switch (switch 1, 2 or 3) I can ping 192.168.13.1 (switch 4). When I try and ping switch 4 from over the MPLS I am unable to. If I trace to the switch I see it reaches the outside of the MPLS router, but is then unresponsive. The same applies if I try to ping switch 1 on 192.168.13.2. Any of the other IP addresses of switch 1 respond.

The MPLS network is a managed solution to which I have no access. I'm told that the MPLS provider is able to ping switch 1 & switch 4 on the 192.168.13.x addresses from a remote router (192.168.32.2). I have tried from a switch on the same L2 subnet (192.168.32.1) and I don't get a response.

From switch 4 I am able to ping the switch on 1 of it's interfaces (192.168.19.1), but not the interface I mentioned above 192.168.32.1

There are no access lists in place on the switches and no firewalls between the sites.

I have no idea where to start troubleshooting this, and any assistance would be much appreciated.

Thanks,

Neil

[Peter Paluch]

Hi Neil,

A couple of questions and remarks - please go over each of them carefully:

• Your exhibit contains duplicate addresses 192.168.10.2/24, 192.168.11.2/24 and 192.168.12.2/24 indicated by switches 1 and 2. Is it a typo or do these switches indeed have the same IP address? That would be a conflict of IP addresses.
• The 10.1.10.1 address is not indicated in your exhibit but I assume that is configured on some other interface of switch 4.
• I assume that the routed link between switch 1 and switch 4 is using the IP address 192.168.13.0/24 with switch 4 being .1 and switch 1 being .2. Is that correct?
• From switch 4, can you ping 192.168.12.254? (I assume that 192.168.12.254 is the IP address of the MPLS router)
• From switch 4, can you ping 192.168.12.254 if you use the source IP address 10.1.10.1? Use the ping 192.168.12.254 source 10.1.10.1 command to accomplish this.
• When you enter the show ip eigrp topology command on switch 1, do you see the networks 192.168.3.0/24 and 10.1.10.0/x being present? Can you post this output here? You may delete the irrelevant networks from this output.
• Is the EIGRP neighborship between switch 1 and the MPLS router working? Check that with show ip eigrp neighbor command.
• Can you post the EIGRP configuration from switch 1 here please?

Thank you!

Best regards,

Peter

[#3]

Hello Peter - thanks for your response.

Your exhibit contains duplicate addresses 192.168.10.2/24, 192.168.11.2/24 and 192.168.12.2/24 indicated by switches 1 and 2. Is it a typo or do these switches indeed have the same IP address? That would be a conflict of IP addresses.
The 10.1.10.1 address is not indicated in your exhibit but I assume that is configured on some other interface of switch 4.

Apologies - both of these were mistakes in the example addresses I've given. I have since corrected the diagram and original post.

I assume that the routed link between switch 1 and switch 4 is using the IP address 192.168.13.0/24 with switch 4 being .1 and switch 1 being .2. Is that correct?

This is correct.

From switch 4, can you ping 192.168.12.254? (I assume that 192.168.12.254 is the IP address of the MPLS router)

Yes, I can ping the router, and in fact can ping other sites, however not all subnets at other sites. As I breifly explained above, I am able to ping a switch at a remote site on 1 vlan interface, but not others.

From switch 4, can you ping 192.168.12.254 if you use the source IP address 10.1.10.1? Use the ping 192.168.12.254 source 10.1.10.1 command to accomplish this.

As you highlighted the 10.1.10.1 address was incorrect. I have updated the example above, but this address should have been 192.168.13.1. I have since corrected this, but 10.1.10.1 doesn't exist.

This is the only IP address on switch 4 so the result is the same as the above point. It is successful.

I have however tested this on switch 1 using ping 192.168.12.254 source gigabitEthernet0/4 and this fails.

When you enter the show ip eigrp topology command on switch 1, do you see the networks 192.168.3.0/24 and 10.1.10.0/x being present? Can you post this output here? You may delete the irrelevant networks from this output.
Is the EIGRP neighborship between switch 1 and the MPLS router working? Check that with show ip eigrp neighbor command.
Can you post the EIGRP configuration from switch 1 here please?

Heres the eigrp topology output for 192.168.13.0/24. As mentioned above 10.1.10.0 was a typo and doesn't exist. This is a directly connected route however, switch 1 & switch 4 are not exchanging routes via eigrp.

EIGRP-IPv4 Topology Entry for AS(100)/ID(192.168.12.2) for 192.168.13.0/24
State is Passive, Query origin flag is 1, 1 Successor(s), FD is 2816
Descriptor Blocks:
0.0.0.0 (GigabitEthernet0/4), from Connected, Send flag is 0x0
Composite metric is (2816/0), route is Internal
Vector metric:
Minimum bandwidth is 1000000 Kbit
Total delay is 10 microseconds
Reliability is 255/255
Load is 1/255
Minimum MTU is 1500
Hop count is 0
Originating router is 192.168.12.2

I have confired the neighbour relationship is ok. Other routes are currently advertised out of this site. It is a production site and I'm only experiencing issues with the subnet over the routed link.

Here is the eigrp config from switch 1.

router eigrp 100
distribute-list EIGRP_RECIEVE in
network 192.168.0.0 0.0.255.255
passive-interface default
no passive-interface Vlan12
eigrp stub connected summary

I hope the above makes some sense. Anything else that you need, please let me know.

Thanks,
Neil

[Peter Paluch]

Hello Neil,

Thank you for the response!

One single question here: do I understand you correctly that

• pinging 192.168.12.254 from switch 4 works (source IP = 192.168.13.1)
• pinging 192.168.12.254 from switch 1 does not work when using source IP 192.168.13.2

Are these observations correct?

Best regards,

Peter

[#5]

Yes - that is correct.

SW_SU-04#ping 192.168.12.254 source 192.168.13.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.254, timeout is 2 seconds:
Packet sent with a source address of 192.168.13.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms

SW_SU-01#ping 192.168.12.254 source 192.168.13.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.254, timeout is 2 seconds:
Packet sent with a source address of 192.168.13.2
.....
Success rate is 0 percent (0/5)

[Peter Paluch]

Hi Neil,

Thank you! We will probably need to cooperate with the MPLS service provider now as we will need some output from the MPLS router, specifically:

• show ip route 192.168.13.1
• show ip route 192.168.13.2
• show ip cef 192.168.13.1 internal
• show ip cef 192.168.13.2 internal

Can you ask your MPLS service provider to send you these information? Assuming that the MPLS router may already use some VRF tables for you, tell the MPLS SP to add appropriate VRF references where necessary when using the commands above.

Best regards,

Peter

[#7]

Thanks Peter - appreciate the help.

The MPLS provider are insistant that the problem does not lie with them. None the less I shall request this information. Please bear with me as it could take a little while for them to come back to me.

Cheers,

Neil

[Peter Paluch]

Neil,

I understand why they insist In many cases, it is really the customer's fault. However, this does not seem to be the case here: you are clearly advertising the 192.168.13.0/24 network to the MPLS router, and the router is capable of speaking to just one of two IP addresses inside this network. That means that it has (at least partial) knowledge about that network.

Please double check the Switch1 configuration for any clues of ACL, VLAN ACL (VACL = vlan filter-map), MAC ACLs, anything that could cause the packets between Switch1 and the MPLS router to fail.

You may also want to verify the two commands on Switch 1:

show ip cef exact-route 192.168.13.2 192.168.12.254

show ip cef exact-route 192.168.12.254 192.168.13.2

Please post the results here. Thank you!

Best regards,

Peter

[#9]

Here's the output of the requested commands:

SW_SU-01#show ip cef exact-route 192.168.13.2 192.168.12.254
192.168.13.2 -> 192.168.12.254 => IP adj out of Vlan12, addr 192.168.12.254
SW_SU-01#show ip cef exact-route 192.168.12.254 192.168.13.2
192.168.12.254 -> 192.168.13.2 => receive

I've also attached a copy of the running config. I have truncated the output where appriopriate.

I'm still waiting on the ISP. Any questions, please let me know.

Neil

[Peter Paluch]

Hello Neil,

The show ip cef commands produced correct results - the direction of packets towards 192.168.12.254 is correctly out the SVI for Vlan12, and in return path, the responses are supposed to be received.

After reviewing your configuration, I am slightly surprised by the HSRP running on the SVI VLAN12. Are there any PCs connected to VLAN12? Also, what is the second device that also runs HSRP? Can you perhaps update your exhibit so that it contains these details if it does not already? Thank you!

Best regards,

Peter

[#11]

Apologies for not mentioning HSRP sooner, however I didn't want to 'muddy the water' by mentioning it in my first post.

The 2nd device particiapting in HSRP is Switch 2. This is also a 3560.

You can see from the config that the virtual address for each of the 3 vlans with HSRP configured is .1.

Vlan12 is primarily a server vlan but also has the managed MPLS router sitting on it.

I have updated the diagram as there is also a secondary router. My apologies if this has been counter productive, but hopefully you can appreciate why I didn't want to bombard people with too much information.

The second router is the reason why the switches are configured as stubs. I do not wish to advertise learned routes out of the secondary MPLS router.

[Peter Paluch]

Neil,

Don't worry about not revealing the entire detail in the first take. I can understand that very well.

However, I am now trying to make sense of the routing situation you have present in your network. Let me think aloud and correct me whenever necessary:

• Switch 1 knows about the 192.168.13.0/24 as a directly connected network, and advertises it to both MPLS router 1 and MPLS router 2 as they are both in VLAN12. Correct?
• Switch 2 knows about the 192.168.13.0/24 network via EIGRP. However, it does not tell about it to MPLS router 2 because of the EIGRP stub setting on Switch 2. Correct?
• How is the MPLS router 2 used? Is it indeed only a backup router, or is it being used in some kind of load balancing scenario?

Best regards,

Peter

[#13]

Switch 1 knows about the 192.168.13.0/24 as a directly connected network, and advertises it to both MPLS router 1 and MPLS router 2 as they are both in VLAN12. Correct?

This is correct. Switch 1 has a eigrp neighbour relationship with both MPLS router 1 & 2.

Switch 2 knows about the 192.168.13.0/24 network via EIGRP. However, it does not tell about it to MPLS router 2 because of the EIGRP stub setting on Switch 2. Correct?

Again, I believe this to be correct and that is certainly the intention. Switch 1 also has a neighbour relationship with switch 2. Switch 2 has the summary route for this network.

How is the MPLS router 2 used? Is it indeed only a backup router, or is it being used in some kind of load balancing scenario?

The secondary router is soley for backup purposes.

Cheers,

Neil

[#14]

Hello Peter,

Peter Paluch wrote:
Hi Neil,
Thank you! We will probably need to cooperate with the MPLS service provider now as we will need some output from the MPLS router, specifically:
show ip route 192.168.13.1
show ip route 192.168.13.2
show ip cef 192.168.13.1 internal
show ip cef 192.168.13.2 internal
Can you ask your MPLS service provider to send you these information? Assuming that the MPLS router may already use some VRF tables for you, tell the MPLS SP to add appropriate VRF references where necessary when using the commands above.
Best regards,
Peter

I have had the output from the requested commands back from the ISP. Is there anything in particular I should be looking for?

The ip route commands both came back identical.

Show ip cef is also identical, however they only provided this for 192.168.13.1.

Any help would be appreciated.

Neil

[Peter Paluch]

Neil,

Can you post the information here in its entirety? Can they complete the show ip cef outputs for the other IP address as well?

Best regards,

Peter