Strange Problem

victor_87 · ‎10-08-2008

I have a Cisco 6509 Chasis with SUP 720-B MSFC. I have around 30 Vlans on the 6500, each having address in the 172.16.0.0 series with mask 21, on the 6500 i have a default route to my Cisco ASA for internet connectivity.

Everything was fine until today morning when i started seeing a strange problem,

i received a call saying users are unable to access the Mail server which is located in vlan1 and is directly connected to the Gi port on the 6500, I tried to access the mail server from another PC also in the same subnet as the mail server, i could access it intermittently.

I then continously Pinged to the server and This is what i could see

Reply from 172.16.1.11: bytes=32 time<1ms TTL=64

Reply from 6.0.0.2: TTL expired in transit.

Reply from 172.16.1.11: bytes=32 time<1ms TTL=64

why is the request going to some 6.0.0.2 which is somewhere in the internet i suppose, when 172.16.... is a direclty connected route in the Routing table on my 6500. and moreover i am facing this problem with systems in the same Vlan where there is no routing involved, it's mere switching.

i dont suspect the SUP,. as i have a redundant SUP to which i switched over and checked.

i dont suspect the mail server because i am facing the same problem with all other PC's in the same Vlan

Internet Connectivity to all the PC's is perfectly alright.

Please get me out of this trouble, i am unable to explain anything to the Customer.

victor_87 · ‎10-08-2008

Bloody Hell i solved the problem and it really is strange.

I am still wondering how the hell this can happen, i has come across this same thing once earlier but i forgot abt it as it didn't cause much problem then.

So i'll tell you people what happened and if someone could please add some logic to it, i'll be thankful.

All traffic from all Vlans come onto my Cisco 6509, it has a default route to my firewall, the firewall has a default route to a Link load balancer(has two ISP links), the load balancer has default routes to my two routers,

Cisco 6509-->cisco ASA------> Linkproof----->normal L2 Switch(2950)-------> Router.

So the worst thing is i have a Patch chord attached to that 2950(Public Switch) to a port in Vlan 1 on the 6509. I did that so that i can use public Ip's on my PC whenever i wanted to By pass the firewall rules. (pretty nasty huh).

Now tell me what in god's name was happening, the PC's in vlan 1 have private addresses (172.16.0.X/21) gateway 172.16.0.2 (vlan interface IP on the Cisco 6509 ). but sometimes when i traceroute to the internet

the first hop actually has to be my gateway, but it directly goes to the Router.

How the hell can that happen

If someone didn't understand, notify, i'll explain in detail.

If u did get me, please help my head get the most stupid routing logic involved.

Giuseppe Larosa · ‎10-08-2008

Hello Victor,

you have realized a L2 bypass in vlan 1 from cat6509 to Router via public switch C2950.

a)So: Router interface is connected to a C2950 port in vlan1

B) Router has ip proxy-arp enabled on its internal interface.

When the PC in vlan1 tries to resolve its GW address 172.16.0.2 Router can try to answer with its own MAC address.

If PC builds the association IP 172.16.0.2 to Router's interface mac address it will send traffic to router bypassing all the L3 chain.

to verify this interpretation

on Router

do sh ip interface type x/y

look for the line about proxy-arp and check if it is enabled

to disable it

int type x/y

no ip proxy-arp

on PC you can verify

on a OS shell

arp -g

verify the entry for ip 172.16.0.2

Use a separate different vlan for management and for this bypass vlan.

Move the servers in a third vlan "DMZ"

Hope to help

Giuseppe

victor_87 · ‎10-09-2008

Your interpretation does seem logical and i'll check on the Proxy-arp configuration tomorrow.

ok even if the PC bypasses the L3 chain, there is a reverse route on the router saying that 172.16.0.0 network is on its inside interface itself, i agree that there is no outbound entry in the firewall for this traffic, so it'll drop it but, why the traffic is trying to go to some place far away in the internet.

Anyway thankyou very much.