2 Vlans Stopped Communicating

Unanswered Question
Apr 30th, 2008
User Badges:

Our School District has 25 buildings and each have it's own vlan assigned to it. Recently one of our building vlans, 111, just stopped communicating with another, 157, but does communicate with the remaining 23 vlans. Vlan 157 is not able to communicate with vlan 111 but can communicate with the other 23 vlans.


We have a 6509 sup720 running CatOS with 3500 series edge switches.


Any suggestions on where to start looking would be greatly appreciated.


Cathy Perry

WWCSD Tech Group

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Jon Marshall Wed, 04/30/2008 - 12:21
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Cathy


Where are the L3 interfaces for the vlans - are they on the 6500 switch ?


The 3550 edge switches, do they have multiple vlans on each switch and are they connected back via trunks to the 6500.


Where are you trying to connect from / to ie. are you on a PC in vlan 111 trying to connect to a PC in vlan 157 and are these on different switches.


Jon

ccoperryc Wed, 04/30/2008 - 17:39
User Badges:

Jon,


The L3 interfaces for these vlans are on the 6500 switch. The edge switches for Vlan157 have multiple vlans and trunk back to the 6500. The edge switches for Vlan111 have been manually pruned with the assistance of Cisco TAC Support last summer and also trunk back to the 6500.


To answer your last question we actually tried to ping from a server in either Vlan to the server in the other Vlan. Pinging in either direction fails. Both servers are connected directly to the 6500 by trunk ports.


Thank you,


Cathy

Jon Marshall Thu, 05/01/2008 - 00:44
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Cathy


What exactly do you mean when you say servers are trunked. If you mean trunked as in cisco trunk then which vlans are they members of ?


Or do you mean trunked in the cisco etherchannel sense ?


If both servers are connected into the 6500 and the routing is done on the 6500 then we can probably rule out an issue with the access-layer switches.


Do you have any filtering with access-lists.


What are the IP address/subnet mask/default-gateway details for your 2 servers.


Jon

ccoperryc Thu, 05/01/2008 - 07:48
User Badges:

Jon,


Each server in our network has a dedicated port on the 6500 for it's individual Vlan.


You are correct, both servers connect directly to the 6500 and routing is done on the 6500.


This is the only access list filtering I have been able to locate on our router.


Extended IP access list 101

10 deny tcp any any eq 5554

20 deny tcp any any eq 9996

30 permit tcp any any eq www

40 deny tcp any any eq 445


Vlan 111 10.91.72.4 255.255.252.0 10.191.72.2


Vlan 157 10.91.48.4 255.255.252.0 10.191.48.2


Cathy

Jon Marshall Thu, 05/01/2008 - 09:06
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Cathy


Do you know if/where the access-list 101 is applied ?


Jon

ccoperryc Thu, 05/01/2008 - 09:36
User Badges:

Jon,


I believe this is what you are asking me:


The access-list 101 in my previous response came from the MSFC on the 6509. I ran the sho access-lists command to get it.


MSFC#sho access-lists

Extended IP access list 101

10 deny tcp any any eq 5554

20 deny tcp any any eq 9996

30 permit tcp any any eq www

40 deny tcp any any eq 445


Cathy

Jon Marshall Thu, 05/01/2008 - 09:37
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Sorry Cathy, i didn't explain myself very well.


Can you post the output of a


sh run int vlan 111

sh run int vlan 157


Jon

ccoperryc Thu, 05/01/2008 - 10:53
User Badges:

Jon,


Thank you for your patience with me.


Here it is:


MSFC#sho run int vlan 111

Building configuration...


Current configuration : 132 bytes

!

interface Vlan111

description Stev-Instr

ip address 10.91.72.2 255.255.252.0

no ip redirects

ip pim sparse-mode

ip cgmp

end


MSFC#sho run int vlan 157

Building configuration...


Current configuration : 132 bytes

!

interface Vlan157

description Adam-Instr

ip address 10.91.48.2 255.255.252.0

no ip redirects

ip pim sparse-mode

ip cgmp

end


This is as it has always been.


Cathy

Jon Marshall Thu, 05/01/2008 - 10:58
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Cathy


No problem, i just wanted to confirm the access-list wasn't applied to these vlan interfaces. Sometimes these problems can tke a while to sort out and can take quite a few questions.


The other thing - in a previous post you said servers ip addresses/subnet mask/DG were


Vlan 111 10.91.72.4 255.255.252.0 10.191.72.2


Vlan 157 10.91.48.4 255.255.252.0 10.191.48.2


Are the default-gateways typos ie.


10.191.72.2 should be 10.91.72.2


and


10.191.48.2 should be 10.91.48.2


Jon

ccoperryc Thu, 05/01/2008 - 11:25
User Badges:

Sorry Jon,


Yes, 10.91.72.2 and 10.91.48.2 are the appropriate gateways.


I was hopping in and out of switches and do this sometimes.


Cathy

Jon Marshall Thu, 05/01/2008 - 11:37
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Cathy


Okay can we try a few things


1) From the server in vlan 111 can you ping vlan 157 interface - result ?

2) From server in vlan 157 can you ping vlan 111 interface ?

3) From server in vlan 111 can you ping a server in a different vlan ?

4) Same as 3 for server in vlan 157.

5) Can you post a "sh ip route" from the 6500.

6) Can you post the interface vlan configuration off the 6500.


Edit - which module(s) are the servers connected into and which IOS version are you running.


Apologies for requesting all this info but there is nothing obvious (at least to me ! )


Jon

ccoperryc Fri, 05/02/2008 - 07:08
User Badges:

Jon,


I agree there is nothing obvious, especially since there have been no configuration changes to the 6500 in at least a month.


1) From the server in vlan 111 can you ping vlan 157 interface - result ?

Yes


2) From server in vlan 157 can you ping vlan 111 interface ?

No


3) From server in vlan 111 can you ping a server in a different vlan ?

4) Same as 3 for server in vlan 157.

Yes, to both 3 & 4


5) Can you post a "sh ip route" from the 6500.


“sh ip route” did not give me the response I expected to see. What information

should I see when I run this for you?


6) Can you post the interface vlan configuration off the 6500.


#module 9 : 16-port 1000BaseX Ethernet

set vlan 16 9/1-2,9/11

set vlan 109 9/3,9/5,9/16

set vlan 111 9/14 (Bldg Server)

set vlan 113 9/7

set vlan 115 9/15

set vlan 139 9/6

set vlan 141 9/4

set vlan 147 9/13

set vlan 153 9/12

set vlan 157 9/9-10 (IE filtering Server & Bldg Server)

set vlan 888 9/8

set port name 9/3 Nautilus

set port name 9/8 Lincoln-Vandenberg

set port name 9/16

set cdp disable 9/16

set udld enable 9/3

set trunk 9/3 on dot1q 1-4094

set trunk 9/4 on dot1q 1-4094

clear trunk 9/8 2-15,17-130,133-144,147-4094

set trunk 9/8 on dot1q 1,16,131-132,145-146

set trunk 9/9 off dot1q 1-4094 (this config questionable)

set spantree portfast 9/1-16 disable



Edit - which module(s) are the servers connected into


Server in Vlan 111 is connected to 9/14 - Building Server


Server in Vlan 157 is connected to 9/9 - IE Filtering Server 9/10 - Building Server


Which IOS version are you running.


Sup720 is running cat6000-sup720k8.8-5-9


MSFC "bootflash:c6msfc3-psv-mz.122-17d.SXB8"

MSFC3 Software (C6MSFC3-PSV-M), Version 12.2(17d)SXB8, RELEASE SOFTWARE (fc2)


Jon Marshall Fri, 05/02/2008 - 07:26
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Cathy


You ran the "sh ip route" on the supervisor but you are running in hybrid mode so you need to run it from the MSFC.


Can you


1) Post the output of a "sh module"


2) Log on to the MSFC and post the output of a "sh ip route"


3) On the MSFC post the output of the running config - "sh run"


Jon

ccoperryc Fri, 05/02/2008 - 09:50
User Badges:

Jon,


Attached are the outputs you have requested.


Again I would like to thank you for your patience. As you have probably already figured out I am new to trouble shooting so thank you.


Cathy



Attachment: 
Jon Marshall Fri, 05/02/2008 - 10:29
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Cathy


Don't thank me just yet as there is nothing obviously wrong in your configs.


One thing


set vlan 111 9/14 (Bldg Server)


set vlan 157 9/9-10 (IE filtering Server & Bldg Server)


Is the Bldg server the same server in both set commands. ie does it have 2 NIC's, one in vlan 111 and one in vlan 157.


If so can you print off the routing table off this server


Jon

lamav Fri, 05/02/2008 - 10:38
User Badges:
  • Blue, 1500 points or more

"Jon, there have been no configuration changes to the 6500 in at least a month."


You sure about that??


This is from the output of the sho run on the MSFC that you just sent us:


MSFC#sho run

Building configuration...


Current configuration : 9610 bytes

!

! Last configuration change at 22:55:39 EDT Tue Apr 29 2008

! NVRAM config last updated at 22:26:23 EDT Tue Apr 29 2008


The configuration change was made the night before you started this thread. Sound interesting to you? Not only was a config change made, it was saved, too. So, if the change is what caused this problem, rebooting the switch may not help you.


VICTOR


ccoperryc Fri, 05/02/2008 - 11:22
User Badges:

Victor,


Thank you for bringing this to my attention. This change had slipped my mind since it did not correct the problem. There had been no other changes in the switch since April 4.


The problem had started before this configuration change. When making the change did not correct the problem I decided to start this thread.


Hope this clears up the confusion.


Thank you


Cathy

ccoperryc Fri, 05/02/2008 - 11:18
User Badges:

Jon,


These are 3 separate servers


vlan 111 9/14 goes to Stevenson MS server

vlan 157 9/10 goes to Admams MS server

vlan 157 9/9 goes to Adams IE filter server.

Myself and our Network Administrator have checked DNS/DHCP Mgmt to ensure each has the proper gateway loaded if this is what you are asking for. Vlan 111 has a gateway of 10.91.72.2 Vlan 157 has a gateway of 10.91.48.2.


Cathy

Jon Marshall Fri, 05/02/2008 - 12:14
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Cathy


Have checked your configs and there is still nothing obvious.


Are both servers in the 157 vlan unable to connect to vlan 111 server.


What do your arp/mac-address tables/cef tables look like when you try to communicate between the 2 vlans.


I'll have a think over the weekend and see if anything else comes to mind.


Jon

tj.mitchell Wed, 05/07/2008 - 07:58
User Badges:
  • Bronze, 100 points or more

Do you have copy of the configuration before the change was made?

ccoperryc Wed, 05/07/2008 - 10:23
User Badges:

TJ,


Not sure which change you are referring to. I have several configurations of both the Sup720 and MSFC. If you are referring to the change I made on 4/29/08 all I did was remove a No Ip Unreachables config from vlan 111 in the MSFC.


I always take a fresh copy of the configs before I make any changes to them. Except when it is what I believe to be a small change as this one.


Please let me know which configs you would like to see.


Thank you


Cathy Perry

ccoperryc Fri, 05/02/2008 - 14:02
User Badges:

Jon,


Yes, both servers in vlan 157 are not able to ping to vlan 111 server.


As far as arp/mac-address tables/cef tables go I don't have that level of experience / knowledge to know what I would be looking at.


Here's a thought:


As I have been reading the different posts I have been pouring over configurations from different points in time from the 6500 and two things continue to stick out to me:


PortInstanceCost on the Sup 720


and


Control Plane on the MSFC.


Is there any remote possiblity that either of these could be the cause?


These are both in previous configs I have looked through but not the current configuration. I haven't mentioned them since these vlans could communicate with each other for about 3 1/2 weeks after the last time we touched the configs on the sup720.


Thanks and have a great weekend.


Cathy

lamav Fri, 05/02/2008 - 07:31
User Badges:
  • Blue, 1500 points or more

5) Can you post a "sh ip route" from the 6500.


“sh ip route” did not give me the response I expected to see. What information

should I see when I run this for you?


6) Can you post the interface vlan configuration off the 6500.



sh ip route is off of the MSFC, not the supervisor.


And I think Jon is asking for the interface configurations for both vlans from the MSFC, not the module.

lamav Fri, 05/02/2008 - 10:36
User Badges:
  • Blue, 1500 points or more

"there have been no configuration changes to the 6500 in at least a month."


You sure about that??


This is from the output of the sho run on the MSFC that you just sent us:


MSFC#sho run

Building configuration...


Current configuration : 9610 bytes

!

! Last configuration change at 22:55:39 EDT Tue Apr 29 2008

! NVRAM config last updated at 22:26:23 EDT Tue Apr 29 2008


The configuration change was made the night before you started this thread. Sound interesting to you? Not only was a config change made, it was saved, too. So, if the change is what caused this problem, rebooting the switch may not help you.


VICTOR


!


t814687 Thu, 05/01/2008 - 12:07
User Badges:

Cathy, can you clear arp and if that does not help, reload the 6509?

have you tried that?



Sorry guys for interjecting, I do not see that the setup is that complicated and the notion that the issue just happened by itself gave me an idea of a software glitch that could be fixed by reloading a switch.

Agree with Jon, you guys in a better position to see a feasibility for the reload. Sometimes it's a quick fix.


One more thing is to look through the logs of 6509 to make sure there is not critical errors there....


-serg

ccoperryc Thu, 05/01/2008 - 12:16
User Badges:

Jon,


I did run the clear arp command last night which did not help.


I like the idea of a reload except it is very difficult to schedule with night classes in the district. I will see what we can do.


Cathy

Jon Marshall Thu, 05/01/2008 - 12:19
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Cathy


Different person posted about clearing arp and reloading.


Problem with reloading is if it fixes it you don't actually know what fixed it so if it happens again you are non the wiser and you have to reload again. Depends on how much downtime you can get on your switch. And bear in mind that a reload can sometimes bring a different set of problems.


But not saying you shouldn't do it. You are in the best position to judge.


Jon

bs6825 Thu, 05/01/2008 - 12:41
User Badges:

Cathy,


Check the trunk links to make sure that both of the vlans in question are allowed on the trunks. Also check the VTP configuration revisions to make sure they are consistent. Run the show vlan command on each switch, make sure both vlans are on both switches. It does not appear to be a routing (layer 3) problem since they are functioning properly with other vlans and you have ruled out an ACL causing the issue. Appears to be a layer 2 issue.


good luck

Jon Marshall Thu, 05/01/2008 - 12:45
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Bruce


This was my thought too but Cathy says that servers are connected directly into the 6500 switch. So i thought it might just complicate things to start checking trunk links to access-layer switches etc.


But might be worth a try.


Jon

ccoperryc Thu, 05/01/2008 - 18:12
User Badges:

Jon,


You are correct both servers connect directly to the 6500. I have looked at the module ports for both servers several times and can confirm that both ports are set to allow all vlans through.


As I have been reading the different posts I have been pouring over configurations from different points in time from the 6500 and two things continue to stick out to me:


PortInstanceCost on the Sup 720


and


Control Plane on the MSFC.


Is there any remote possiblity that either of these could be the cause?


I will run the trace routes on Friday for you as requested in an earlier post.


Thanks to all,


Cathy

lamav Thu, 05/01/2008 - 19:12
User Badges:
  • Blue, 1500 points or more

Cathy/Jon:


Not to be a budinski, but how about Cathy just post the entire 6500 switch config so we dont have to keep guessing what the configs are?


Just a thought instead of shooting in the dark...


Thanks


Victor

lamav Mon, 05/05/2008 - 07:36
User Badges:
  • Blue, 1500 points or more

Cathy:


Is it fixed?


VL

ccoperryc Mon, 05/05/2008 - 10:06
User Badges:

Still having issues.


Thank you for asking.


Cathy

t814687 Mon, 05/05/2008 - 10:09
User Badges:

is it feasible for you guys do this after hours:


shut

no shut


on the VLAN interfaces you are having issues.

this should clear the cef tables....


-serg

ccoperryc Mon, 05/05/2008 - 10:17
User Badges:

Serg,


That could be a definate possibility I would have to schedule it through our Network Administrator.


Let you know.


Thanks


Cathy

lamav Mon, 05/05/2008 - 14:05
User Badges:
  • Blue, 1500 points or more

Cathy, sorry to hear it.


If you decide to reboot the switch, do not forget to do a 'wr mem" on the MSFC so that you dont lose any unsaved configurations.


After that, do the reboot on the supervisor and the MSFC.


Thanks


VL

ccoperryc Wed, 05/07/2008 - 04:50
User Badges:

Serg,


Id the shut no shut on the two vlans having problems. No change in the status.


Thanks for the suggestion.


Cathy

t814687 Wed, 05/07/2008 - 10:29
User Badges:

Cathy,

from configuration perspective the configs of 6509 look pretty normal and if there is no any acl involved the directly attached subnets should talk to each other. Unless people have any other ideas I would schedule a reload of the box at some time after hours.

Think about that...


-serg

ccoperryc Thu, 05/08/2008 - 08:17
User Badges:

Victor,


Thank you for the support and humor.


We have discussed and agree that we are at the point of a reboot and will be scheduling this soon.


Cathy

lamav Thu, 05/08/2008 - 08:43
User Badges:
  • Blue, 1500 points or more

OK, cool. Let us know what happens, for sure.


Victor

dmooreami Thu, 05/08/2008 - 09:31
User Badges:

Humm, I had a very similiar problem. see my post here:

http://forum.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Network%20Infrastructure&topic=LAN%2C%20Switching%20and%20Routing&topicID=.ee71a04&fromOutline=&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.2cc07deb


Short was one vlan could not even ping its default l3 gateway. had to remove the interface in the IOS and re-apply it.


All L3 Vlans on the switch could ping the problem Vlans L3 gateway. It was like the marriage between the L3 and L2 in the cat where broken and not talking. I onl have Sup2's, not fancy 720's like you do. :)



ccoperryc Mon, 05/12/2008 - 06:21
User Badges:

Found something very interesting last Thrusday morning after getting this post in the Cisco Forum.


I can ping the gateway for Vlan 157 (AMS) from the server in vlan 111(SMS) with no problem. (ping 10.91.48.2 from 10.91.72.4 server)


I cannot ping the gateway for Vlan 111(SMS) from the server in Vlan 157 (AMS). (ping 10.91.72.2 from 10.91.48.4 server)


Any thoughts?


Thank you


Cathy


dmooreami Mon, 05/12/2008 - 06:27
User Badges:

Sounds like an access-list blocking Vlan111 to Vlan157, but V157 is allowed into V111.


Don't know if any firewalls are involved that might come into play also

lamav Tue, 05/20/2008 - 07:06
User Badges:
  • Blue, 1500 points or more

Cathy:


Whatever happened with this?


Victor

ccoperryc Tue, 05/20/2008 - 09:13
User Badges:

Victor,


We did a reset of the 6500 a week ago this past Friday and it did not fix the problem. Boy did it uncover and even larger problem. Since we uploaded the config from a pcmcia card when the Sup720 went in it had the wrong boot file set and the router didn't come back up. Called Cisco tech at 12:00 am and was back online by 12:30 am


Anyway, I have also been communicating with someone I know at our Regional School District Support office and he has come to the same conclusion as was previously suggested in this posting, Posted by: dmooreami - May 8, 2008, 10:31am PST,

and that was to delete the problem vlans from the router and re-add them. We haven't had a window of opportunity to do this yet.


Sorry for such a long delay in responding to let you all know if the reset worked or not. Trying to catch up on some other tasks that have fallen behind.


I have all the configs I need just not quite sure what all is removed once I delete the vlan. This is what is currently set:


interface Vlan111

description Stev-Instr

ip address 10.91.72.2 255.255.252.0

no ip redirects

ip pim sparse-mode

ip cgmp


Do I just re-add it to the router and configure as above?


Cathy

dmooreami Tue, 05/20/2008 - 09:17
User Badges:

easy:


No interface Vlan111


Leave your L2 section alone.


Wait about 10 mins, paste back in your config.


interface Vlan111

description Stev-Instr

ip address 10.91.72.2 255.255.252.0

no ip redirects

ip pim sparse-mode

ip cgmp


No need to reboot the switch


Only applies to hybrid mode 6500's runing CATOS. If running Native, this won't fix anything.


ccoperryc Tue, 05/20/2008 - 09:23
User Badges:

Sounds easy enough. I will have to schedule and evening time window to do this and with the Holiday weekend coming probably not until next week sometime. Most likely a week from this Thursday evening.


Thanks,


Cathy

Actions

This Discussion