×

Warning message

  • Cisco Support Forums is in Read Only mode while the site is being migrated.
  • Cisco Support Forums is in Read Only mode while the site is being migrated.

Routing Mystery - where is it going?

Unanswered Question
Dec 18th, 2002
User Badges:

I have a mystery and before I call Leonard Nimoy in to do a "In Search Of...." episode thought I post here.


PROBLEM:

I'm converting a remote site from ISDN (At the moment using Ascend Pipelines) and converting the site to Point-to-point using a 1760 Cisco on the remote end and a 7204 vxr on the other. After conversion, everything works except DHCPOFFER doesn't hit the workstation. EVERYTHING works except that. Ping/Telent/Http..blahblah. I can see DHCPDISCOVER packet hit the DHCP server, I can see the DHCP server sending a DHCPOFFER but workstation doesn't get it. If I put a different IP subnet for this location the offer goes through. Routing problem right?


SETUP:

[workstation] - IP 172.22.36.7

|

[1706 Cisco] - E0=172.22.36.1 S0=172.22.136.1 (ip helper on E0)

|

[7204 VXR Cisco] - S0=172.22.136.2 E0=172.22.1.8 (Ip helper on E0)

|

[2948G-L3 Cisco] - 172.22.1.1

|

[DHCP Server] - 172.22.1.50


TROUBLESHOOTING THUS FAR:

1. Check routes on all routers/servers/workstations

2. Clear routers on everything. We are using EIGRP.

3. Enable Show Debug on all routers

with this I see the DHCPDISCOVER hit everything,

I don't see an DHCPOFFER hit the 172.22.1.1 on the way back

4. Put a Sniffer on DHCP server and on the Workstation ends

- Sniffer shows its heading to the right place

5. Enable DHCP debug on the server, shows everything is fine except it can't send DHCPOFFER.

6. If I go to a workstation after converting the site, and it still has an IP, the dhcpserver can ping the workstation no problem. Once I release the IP it can't get another one.

7. I have 7 total sites, 4 sites are working - three are not, using the exact same config, except different ips.

8. I removed all sites from the 7200VXR and cleared all routes and then attempt switch over but still had problem. I was thinking something was advertising the wrong route.


SUGGESTIONS?

I know I can get it to work if I change the IP subnet. Really don't want to do this, need to figure out issue, who knows when and where it will rear its ugly head.


Any help will be great!

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
prafuljaded Wed, 12/18/2002 - 14:01
User Badges:

Is your workstation directly connected to e0 of the router or there is some L3 device between the two. If yes,configure ip helper-address on the port facing the workstation. Also your Router/L3 should have path to DHCP Server subnet.

bleucube Wed, 12/18/2002 - 14:12
User Badges:

No Workstation is connected from the Router on the remote end, goes directly into a stupid hub.


At one point in my troubleshooting i had Helper address on every interface.

Also, I made sure the DHCP is setup properly with option 3 (router) to the correct IP. Like I said, I have four sites working and the setup is the same except IP's. I had three other set of eyes check the configs and to compare and yep - all the same.




a.balian Wed, 12/18/2002 - 14:51
User Badges:

My friend :

You have a very interesting problem. I suggest you look into the scope of your DHCP server. Use netstat -a to see if the port 67 68 are listening. Your IP variation and DHCP scope have something to do with your problem that you anly can solve it.


Good luck


lgijssel Thu, 12/19/2002 - 02:58
User Badges:
  • Red, 2250 points or more

There is one sneaky issue with cisco routers that might cause this type of problem. Most important questions:

- To what range did you change the IP's to get it to work?

- How do you route from your remote sites?

My assumption is that you are using supernet routes from the remote locations, and you have configured no ip classless on the 1760 router.

Changing to an ip-network outside 172.22 will result in a working situation as the supernet route is now the best match. Without ip classless, the packet will be discarded by the 1760.

Effectively all you should do, if this is the problem, is to enable ip classless.


bleucube Thu, 12/19/2002 - 04:07
User Badges:

- I changed the range to our test subnet which is 172.22.15.0


- I use EIGRP from my remote sites, however the existing sites that I need to transfer over to T1 are using RIP (not cisco routers) and I have RIP setup on core router (172.22.1.1). During troubleshooting I did remove the sites using RIP from the core router and clear ip routes.


- Actually have IP CLASSESS on ... here's the config. Very short and simple. Good guess tho.


version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname #######

!

enable password 7 #######

!

no ip subnet-zero

no ip domain lookup

!

interface FastEthernet0/0

ip address 172.22.36.1 255.255.255.0

ip helper-address 172.22.1.50

speed auto

!

interface Serial0/0

ip address 172.22.136.1 255.255.255.0

no fair-queue

!

router eigrp 200

network 172.22.0.0

no auto-summary

no eigrp log-neighbor-changes

!

ip classless

ip route 0.0.0.0 0.0.0.0 172.22.136.2

no ip http server

!

line con 0

password 7 #######

login

line aux 0

password 7 #######

login

line vty 0 4

password 7 #############

login

!

end

lgijssel Thu, 12/19/2002 - 06:29
User Badges:
  • Red, 2250 points or more

That is a simple config indeed. There should be no problem.

Rereading the posting I noticed that DHCP is sent to the server and arrives there. Therefore, th sole thing that I still can think of is related to another posting. C800 routers with IOS 12.0(x)T are having an identical issue with dhcp. Are all 1760's using the same IOS-version?


To check this you can exchange a working 1760 with a non-working one.


Regards and good hunting,

Leo

bleucube Thu, 12/19/2002 - 08:09
User Badges:

All the 1760's and the 7200 VXR are using 12.2


This indeed is weird. I don't understand how a DHCPDISCOVER packet can tranverse the route back to the DHCP server but the DHCPOFFER disappears out of the DHCP server.


So if you looked at the ascii map above, this is what it looks like:


[DHCP] (a)

|

| ---|hub| (b) --------------- Sniffer

|

[172.22.1.1] (c)


A) The DHCP sends the offer out heading in the right direction

B) The Sniffer see's it going the right way

C) 172.22.1.1 show ip debug UDP never see's the packet


It's hard to imagine its getting lost on the physical line... especially since everything else works all DNS request go to this server too.



uniemeyer Thu, 12/19/2002 - 08:26
User Badges:

Hi,


you explained that you did a Sniffer trace on the server site:


Is the IP address of the ethernet interface of the router 1706 included in the

DHCP DISCOVER message?


Is the DHCP server answering with a DHCP OFFER to the MAC-address

of the 7204 router or is the offer also a broadcast?


Ulrich Marzoli



bleucube Thu, 12/19/2002 - 11:22
User Badges:

1) No, just the address of the DHCP server

2) Answer is a broadcast I believe, Need to look up the sniff log. Will respond in a few... need to fire up laptop

msingla Thu, 12/19/2002 - 09:43
User Badges:

Hi,

Strange problem u have .Please do send the other side configuration so that problem can be analyzed in detail.....

Regards,

Munit

bleucube Thu, 12/19/2002 - 10:41
User Badges:

Here's the 7200 VXR Config. Pretty simple too.


version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname #############

!

enable secret 5 #############

enable password 7 #############

!

no ip subnet-zero

ip cef

!

!

no ip domain lookup

!

!

!

controller T1 1/0

framing esf

clock source internal

linecode b8zs

channel-group 1 timeslots 1-24

description "#############"

!

controller T1 1/1

framing esf

clock source internal

linecode b8zs

channel-group 1 timeslots 1-24

description "#############"

!

controller T1 1/2

framing esf

clock source internal

linecode b8zs

channel-group 1 timeslots 1-24

description "#############"

!

controller T1 1/3

framing esf

clock source internal

linecode b8zs

channel-group 1 timeslots 1-24

description "#############"

!

controller T1 1/4

framing esf

clock source internal

linecode b8zs

channel-group 1 timeslots 1-24

description "#############"

!

controller T1 1/5

framing esf

clock source internal

linecode b8zs

channel-group 1 timeslots 1-24

description "#############"

!

controller T1 1/6

framing esf

clock source internal

linecode b8zs

channel-group 1 timeslots 1-24

description "#############"

!

controller T1 1/7

framing esf

linecode b8zs

!

!

!

!

speed auto

!

interface FastEthernet0/1

no ip address

shutdown

duplex auto

speed auto

!

interface Serial1/0:1

description "#############"

ip address 172.22.141.2 255.255.255.0

!

interface Serial1/1:1

description "#############"

ip address 172.22.161.2 255.255.255.0

!

interface Serial1/2:1

description "#############"

ip address 172.22.136.2 255.255.255.0

!

interface Serial1/3:1

description "#############"

ip address 172.22.156.2 255.255.255.0

!

interface Serial1/4:1

description "#############"

ip address 172.22.146.2 255.255.255.0

!

interface Serial1/5:1

description "#############"

ip address 172.22.131.2 255.255.255.0

!

interface Serial1/6:1

description "#############"

ip address 172.22.151.2 255.255.255.0

!

router eigrp 200

network 172.22.0.0

no auto-summary

no eigrp log-neighbor-changes

!

ip classless

ip route 0.0.0.0 0.0.0.0 172.22.1.1

no ip http server

!

!

gatekeeper

shutdown

!

!

line con 0

password 7 #############

login

line aux 0

password 7 #############

login

line vty 0 4

password 7 #############

login

!

!

end

msingla Thu, 12/19/2002 - 12:43
User Badges:

Hi,

Thanx for sending the configuration.I analyzed everthing nothing is faulty.Can you tell which dhcp server u are using most probably i think windows and please do write the source address u are getting of DHCP discover messages in sniffer and moreover the other any traces from sniffer.

Lets try all together where the issue is.

The other way to test is implement the dhcp on cisco IOS temporarily on l3 switch and check now....most probably no routing issues as everything works fine and also as everybody suggested please remove the ip helper command on the dhcp server side as it is no need.

Regards,

Munit


bleucube Thu, 12/19/2002 - 13:37
User Badges:

Thanks for your help!


Our DNS/DHCP runs on Netware 5.1 service pack5 with all the updates thus far, our DHCP server version is 3.1.2c with TCP/IP 5.81o



Here is a copy of the DHCP server debug, its from 172.22.1.50 trying to send the DHCPOFFER to one of the remote sites we are having an issue with - this is after the DHCPDISCOVER hits the DHCP server. It never receives the DHCPACK from the workstation so it keeps sending out the OFFER - finally the workstation saids I GIVE UP and gets the Microsoft IP


2002/12/04 13:48:13 packet received from client <0:50:4:C2:68:AF>.


2002/12/04 13:48:13 Sending BOOTP/DHCP reply to <0:50:4:C2:68:AF> as <172.22.56.1>

85 : Get type:4, IPAddr: 172.22.56.11,

LeaseTime:0,MacIndx:3476,pIP=AC16380B

Using existing IP with subnet AC163800

DetermineLeaseTime: proposed=0, return=86400, pSubnet-

>leaseTime=86400

AMAGet() exit type=4, err=0, addr=172.22.56.11

2002/12/04 13:48:17 packet received from client <0:50:4:C2:68:AF>.


2002/12/04 13:48:17 Sending BOOTP/DHCP reply to <0:50:4:C2:68:AF> as <172.22.56.1>.

92 : Get type:4, IPAddr: 172.22.56.11,

LeaseTime:0,MacIndx:3476,pIP=AC16380B

Using existing IP with subnet AC163800

DetermineLeaseTime: proposed=0, return=86400, pSubnet-

>leaseTime=86400

AMAGet() exit type=4, err=0, addr=172.22.56.11


2002/12/04 13:48:24 packet received from client <0:50:4:C2:68:AF>.

2002/12/04 13:48:24 Sending BOOTP/DHCP reply to <0:50:4:C2:68:AF> as <172.22.56.1>.

105 : Get type:2, IPAddr: 169.254.190.37,

LeaseTime:0,MacIndx:3476,pIP=AC16380B

AMAGet() exit type=2, err=30, addr=169.254.190.37


2002/12/04 13:48:37 packet received from client <0:50:4:C2:68:AF>, IP Address <169.254.190.37>.

107 : Get type:2, IPAddr: 169.254.190.37,

LeaseTime:0,MacIndx:3476,pIP=AC16380B

AMAGet() exit type=2, err=30, addr=169.254.190.37


2002/12/04 13:48:39 packet received from client <0:50:4:C2:68:AF>, IP Address <169.254.190.37>.

129 : Get type:4, IPAddr: 172.22.56.11,

LeaseTime:0,MacIndx:3476,pIP=AC16380B

Using existing IP with subnet AC163800

DetermineLeaseTime: proposed=0, return=86400, pSubnet-

>leaseTime=86400

AMAGet() exit type=4, err=0, addr=172.22.56.11


2002/12/04 13:49:02 packet received from client <0:50:4:C2:68:AF>.


2002/12/04 13:49:02 Sending BOOTP/DHCP reply to <0:50:4:C2:68:AF> as <172.22.56.1>.

133 : Get type:4, IPAddr: 172.22.56.11,

LeaseTime:0,MacIndx:3476,pIP=AC16380B

Using existing IP with subnet AC163800

DetermineLeaseTime: proposed=0, return=86400, pSubnet-

>leaseTime=86400

AMAGet() exit type=4, err=0, addr=172.22.56.11



This is what the SHOW DEBUG IP UDP on the 1706 at the remote end *172.22.51.1 sending to the DCHP Server *172.22.1.50


UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308

UDP: sent src=172.22.51.1(67), dst=172.22.1.50(67), length=328

UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308

UDP: sent src=172.22.51.1(67), dst=172.22.1.50(67), length=328

UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308

UDP: sent src=172.22.51.1(67), dst=172.22.1.50(67), length=328


In one that works it shows a src of 172.22.50, dst:172.22.51.1


I will need to upload sniffer log tomorrow.










namso2 Thu, 12/19/2002 - 14:53
User Badges:

It looks like a DHCP server configuation problem. Windows clients don't respond to BOOTP offers, only DHCP. Make sure the scope for that subnet is configured for DHCP/BOOTP or DCHP only.

bleucube Fri, 12/20/2002 - 09:58
User Badges:

As much as the finger points to DHCP, I don't think it is. The reason why I say this:


1. Had a couple people check the different subnet's and everyone is configured the same.


2. In the subnet I'm having a problem with, If I just change the subnet pool - it works.


The Range Type is set to Dynamic Bootp and DHCP


msingla Fri, 12/20/2002 - 09:35
User Badges:

Thanx for the inf,Iam trying and others too.I am also discusiing the same problem with many people on other groups.If anybody comes out of solution.


bleucube Fri, 12/20/2002 - 10:22
User Badges:

Well, Friday is almost over. Tomorrow me and another Administrator will be SOLVING this, one way or another. I will post the outcome.


Here's is are plan... pretty sketchy but you can probably tell we tried just about everything we can think of...


(Currently RIP is being use on 172.22.1.1 because of the ISDN routers, after today we will not need RIP)


Plan A

Backup Configuration on 172.22.1.1

Remove RIP from 172.22.1.1

Clear all routes

Try Out T1


Plan B

Unplug all remote sites

clear all routes

Try out T1

.......

Plan Z

Change the IP Subnet

Change IP on server

Change IP on printers

Change IP on E0 of both routers 1760 and 7204

Clear all routes

Try out T1

Test everything to see what we broke


See the bad thing about Plan Z, is I know it will fix the issue... whatever that maybe. Who know tho, something else might come up and bite us down the road.




deilert Fri, 12/20/2002 - 10:36
User Badges:
  • Silver, 250 points or more

I do not see the segment for your next hop 172.22.1.1 in your default route in the 7206 config what segment is it on ?

bleucube Fri, 12/20/2002 - 11:00
User Badges:

The 7204 VXR is on the same segment as 172.22.1.1


[workstation] - IP 172.22.36.7

|

[1706 Cisco] - E0=172.22.36.1 S0=172.22.136.1 (ip helper on E0)

| Default route is 0.0.0.0.0 255.255.255.255 172.22.136.2

|

[7204 VXR Cisco] - S0=172.22.136.2 E0=172.22.1.8

| Default route is 0.0.0.0 255.255.255.255 172.22.1.1

|

[2948G-L3 Cisco] - 172.22.1.1

|

[DHCP Server] - 172.22.1.50

bleucube Mon, 12/23/2002 - 04:01
User Badges:

Well Saturday went really well. Wanted to update and close this conversation. I really don't have a defined idea on what was screwing up the routes, but this is what I did.

1. I listened to the traffic again and tracked down all the RIP packets. Found:

a) A couple of Lanrover's using RIP that had old routes. I removed RIP and cleared out the routes. These devices do not need to know about the network, they only need to know about the core router 172.22.1.1

b) I disconnect the three remain ISDN lines from the core router 172.22.1.1

c) I removed RIP from the core router 172.22.1.1, and cleared all routes

2. Went onsite and converted it and everything went great! DHCPOFFER from DHCP made it to the workstation and the Workstation sent back a DHCPACK. Just to test it we excluded the IP the workstation receieved, released all and renewed... sure enough got another IP!!! Yeah. All three took.

So now have all the sites on T1's. I can only surmise something was sending the old routes out.

On a side note, we had one site that couldn't telnet to our Unix service box, found out our unix had old routes defined on it too.

So check yer routes.



Thanks everyone for all your help, and I hope this can help others with same issue.

Actions

This Discussion