2 Vlans Stopped Communicating

[ccoperryc]

Our School District has 25 buildings and each have it's own vlan assigned to it. Recently one of our building vlans, 111, just stopped communicating with another, 157, but does communicate with the remaining 23 vlans. Vlan 157 is not able to communicate with vlan 111 but can communicate with the other 23 vlans.

We have a 6509 sup720 running CatOS with 3500 series edge switches.

Any suggestions on where to start looking would be greatly appreciated.

Cathy Perry

WWCSD Tech Group

[Jon Marshall]

Cathy

Don't thank me just yet as there is nothing obviously wrong in your configs.

One thing

set vlan 111 9/14 (Bldg Server)

set vlan 157 9/9-10 (IE filtering Server & Bldg Server)

Is the Bldg server the same server in both set commands. ie does it have 2 NIC's, one in vlan 111 and one in vlan 157.

If so can you print off the routing table off this server

Jon

[lamav]

"Jon, there have been no configuration changes to the 6500 in at least a month."

You sure about that??

This is from the output of the sho run on the MSFC that you just sent us:

MSFC#sho run

Building configuration...

Current configuration : 9610 bytes

!

! Last configuration change at 22:55:39 EDT Tue Apr 29 2008

! NVRAM config last updated at 22:26:23 EDT Tue Apr 29 2008

The configuration change was made the night before you started this thread. Sound interesting to you? Not only was a config change made, it was saved, too. So, if the change is what caused this problem, rebooting the switch may not help you.

VICTOR

[ccoperryc]

Victor,

Thank you for bringing this to my attention. This change had slipped my mind since it did not correct the problem. There had been no other changes in the switch since April 4.

The problem had started before this configuration change. When making the change did not correct the problem I decided to start this thread.

Hope this clears up the confusion.

Thank you

Cathy

[ccoperryc]

Jon,

These are 3 separate servers

vlan 111 9/14 goes to Stevenson MS server

vlan 157 9/10 goes to Admams MS server

vlan 157 9/9 goes to Adams IE filter server.

Myself and our Network Administrator have checked DNS/DHCP Mgmt to ensure each has the proper gateway loaded if this is what you are asking for. Vlan 111 has a gateway of 10.91.72.2 Vlan 157 has a gateway of 10.91.48.2.

Cathy

[Jon Marshall]

Cathy

Have checked your configs and there is still nothing obvious.

Are both servers in the 157 vlan unable to connect to vlan 111 server.

What do your arp/mac-address tables/cef tables look like when you try to communicate between the 2 vlans.

I'll have a think over the weekend and see if anything else comes to mind.

Jon

[t814687]

Cathy,

The link below can help you with troubleshooting cef issues on the 6509 you have running hybrid mode. They do not specify the mitigation actions but at last this should give you an idea how to look at cef tables and pinpoint inconsistencies.

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00804c5472.shtml

-serg

[tj.mitchell]

Do you have copy of the configuration before the change was made?

[ccoperryc]

TJ,

Not sure which change you are referring to. I have several configurations of both the Sup720 and MSFC. If you are referring to the change I made on 4/29/08 all I did was remove a No Ip Unreachables config from vlan 111 in the MSFC.

I always take a fresh copy of the configs before I make any changes to them. Except when it is what I believe to be a small change as this one.

Please let me know which configs you would like to see.

Thank you

Cathy Perry

[ccoperryc]

Jon,

Yes, both servers in vlan 157 are not able to ping to vlan 111 server.

As far as arp/mac-address tables/cef tables go I don't have that level of experience / knowledge to know what I would be looking at.

Here's a thought:

As I have been reading the different posts I have been pouring over configurations from different points in time from the 6500 and two things continue to stick out to me:

PortInstanceCost on the Sup 720

and

Control Plane on the MSFC.

Is there any remote possiblity that either of these could be the cause?

These are both in previous configs I have looked through but not the current configuration. I haven't mentioned them since these vlans could communicate with each other for about 3 1/2 weeks after the last time we touched the configs on the sup720.

Thanks and have a great weekend.

Cathy

[lamav]

5) Can you post a "sh ip route" from the 6500.

“sh ip route” did not give me the response I expected to see. What information

should I see when I run this for you?

6) Can you post the interface vlan configuration off the 6500.

sh ip route is off of the MSFC, not the supervisor.

And I think Jon is asking for the interface configurations for both vlans from the MSFC, not the module.

[lamav]

"there have been no configuration changes to the 6500 in at least a month."

You sure about that??

This is from the output of the sho run on the MSFC that you just sent us:

MSFC#sho run

Building configuration...

Current configuration : 9610 bytes

!

! Last configuration change at 22:55:39 EDT Tue Apr 29 2008

! NVRAM config last updated at 22:26:23 EDT Tue Apr 29 2008

The configuration change was made the night before you started this thread. Sound interesting to you? Not only was a config change made, it was saved, too. So, if the change is what caused this problem, rebooting the switch may not help you.

VICTOR

!

[t814687]

Cathy, can you clear arp and if that does not help, reload the 6509?

have you tried that?

Sorry guys for interjecting, I do not see that the setup is that complicated and the notion that the issue just happened by itself gave me an idea of a software glitch that could be fixed by reloading a switch.

Agree with Jon, you guys in a better position to see a feasibility for the reload. Sometimes it's a quick fix.

One more thing is to look through the logs of 6509 to make sure there is not critical errors there....

-serg

[ccoperryc]

Jon,

I did run the clear arp command last night which did not help.

I like the idea of a reload except it is very difficult to schedule with night classes in the district. I will see what we can do.

Cathy

[Jon Marshall]

Cathy

Different person posted about clearing arp and reloading.

Problem with reloading is if it fixes it you don't actually know what fixed it so if it happens again you are non the wiser and you have to reload again. Depends on how much downtime you can get on your switch. And bear in mind that a reload can sometimes bring a different set of problems.

But not saying you shouldn't do it. You are in the best position to judge.

Jon

[bs6825]

Cathy,

Check the trunk links to make sure that both of the vlans in question are allowed on the trunks. Also check the VTP configuration revisions to make sure they are consistent. Run the show vlan command on each switch, make sure both vlans are on both switches. It does not appear to be a routing (layer 3) problem since they are functioning properly with other vlans and you have ruled out an ACL causing the issue. Appears to be a layer 2 issue.

good luck