troubleshoot wan connection

[tomredmond]

any and all suggestions welcomed.

I have a remote site on a 1721 router connected to two core routers both 3660's by frame relay pvc's. One link is suppost to be a redundant link to the other. The two links are in two seperate subnets.

I can traceroute and or telnet from the core to the remote router but I cannot ping.

I can also traceroute/telnet from the remote router to the core and beyond but I can't ping.

I can ping from a pc in the core to either of the remote interfaces but a traceroute fails.

The ospf is not being shared and the ip ospf neighbor command say the state is EXSTART or down all though the neighbor id is correct.

I am happy to supply config's answer questions etc

Tom Redmond

[thisisshanky]

I would like to analyze the configs at either ends..

[tomredmond]

they are posted in reply to another response below

Thanks

Tom

[donewald]

Tom,

1. Your OSPF issue might very well be mismatched interface types (timers). Ensure all of your interfaces are same type (point to point (or point to multipoint). (debug ip ospf adjacency on your routers will tell you if you have mismatched interface types (timers))

2. Need to see your frame mapping statements to tell on the ping / traceroute issue.

Hope this helps you,

Don

[tomredmond]

1. all interfaces are point to point

2.the mapping statements are

Serial0.16 (up): point-to-point dlci, dlci 16(0x10,0x400), broadcast

status defined, active

Serial0.17 (up): point-to-point dlci, dlci 17(0x11,0x410), broadcast

status defined, active

the mapping statements from the two core routers are huge but the equivalent dlci's are broadcast status defined, active as above.

hope this sheds some light.

Tom

[mark-obrien]

Tom,

In your remote router, enter "show frame-relay pvc" and verify that your PVCs are up, and that they have been up for a long time. If they have not been up for a while, the PVCs are probably bouncing, which would explain how the OSPF neighbor state never gets beyond EXSTART. Enter the command "show frame-relay lmi" and see if you have a significant number of status timeouts. Either bouncing PVCs or LMI status timeouts can indicate a bad connection between your router and the frame relay switch. You can check your portion with a show interface command for your frame relay interface. Look at the error count. Repeat the command a number of times to see if the errors are incrementing. Errors on your interface can explain bouncing PVCs and LMI status timeouts. If you have a clean line and your PVCs are not staying up or you have LMI status timeouts, there is a problem in your frame relay provider's network.

Also, in the core routers and in the remote router, look at the output from the "show frame-relay map" command and make sure that you have the IP address-to-DLCI mappings that you expect.

I won't point you towards the connections at the core, since they are both behaving the same way, and I assume that those routers have working PVCs to other locations on the same interfaces.

That should give you a start. Good luck.

Mark

[tomredmond]

the output from the remote router of the various commands you suggested are posted below. I think the frame-relay side of thihgs is ok but......

I do have multiple other connections at the core that are fine.

bar#sh frame-relay pvc

PVC Statistics for interface Serial0 (Frame Relay DTE)

Active Inactive Deleted Static

Local 2 0 0 0

Switched 0 0 0 0

Unused 0 0 0 0

DLCI = 16, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0.16

input pkts 3850 output pkts 3015 in bytes 210768

out bytes 190896 dropped pkts 0 in FECN pkts 0

in BECN pkts 0 out FECN pkts 0 out BECN pkts 0

in DE pkts 0 out DE pkts 0

out bcast pkts 2947 out bcast bytes 185696

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

pvc create time 04:05:50, last time pvc status changed 04:05:50

DLCI = 17, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0.17

input pkts 1799 output pkts 4793 in bytes 118261

out bytes 316323 dropped pkts 0 in FECN pkts 0

in BECN pkts 0 out FECN pkts 0 out BECN pkts 0

in DE pkts 0 out DE pkts 0

out bcast pkts 2922 out bcast bytes 184436

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

pvc create time 04:05:51, last time pvc status changed 04:05:51

bar#sh fra

bar#sh frame-relay lmi

LMI Statistics for interface Serial0 (Frame Relay DTE) LMI TYPE = ANSI

Invalid Unnumbered info 0 Invalid Prot Disc 0

Invalid dummy Call Ref 0 Invalid Msg Type 0

Invalid Status Message 0 Invalid Lock Shift 0

Invalid Information ID 0 Invalid Report IE Len 0

Invalid Report Request 0 Invalid Keep IE Len 0

Num Status Enq. Sent 1477 Num Status msgs Rcvd 1477

Num Update Status Rcvd 0 Num Status Timeouts 0

bar#sh frame

bar#sh frame-relay map

Serial0.16 (up): point-to-point dlci, dlci 16(0x10,0x400), broadcast

status defined, active

Serial0.17 (up): point-to-point dlci, dlci 17(0x11,0x410), broadcast

status defined, active

[mark-obrien]

Tom,

How about "show interface s0"?

Mark

[tomredmond]

here you go hope it helps

Bar#sh int s0

Serial0 is up, line protocol is up

Hardware is PowerQUICC Serial

MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation FRAME-RELAY, loopback not set

Keepalive set (10 sec)

LMI enq sent 1974, LMI stat recvd 1974, LMI upd recvd 0, DTE LMI up

LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0

LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE

Broadcast queue 0/64, broadcasts sent/dropped 7929/0, interface broadcasts 792

9

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 05:29:04

Queueing strategy: fifo

Output queue 0/40, 0 drops; input queue 0/75, 0 drops

5 minute input rate 2000 bits/sec, 3 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

14916 packets input, 3721160 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

1 input errors, 0 CRC, 1 frame, 0 overrun, 0 ignored, 0 abort

11918 packets output, 670074 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions

DCD=up DSR=up DTR=up RTS=up CTS=up

Bar#

[mark-obrien]

Tom,

All of this looks clean. Did you do the debugging that Don recommended earlier?

Mark

[tomredmond]

As DONEWALD suggested I used the command DEBUG IP OSPF ADJ, however nothing happened i.e. no messages.

I've included the relevent configs for the three routers involved below. Hopefully someone can spot the problem.

Tom Redmond

Remote router configs

**********************

!

version 12.2

service timestamps debug uptime

service timestamps log uptime

no service password-encryption

!

hostname Bar

!

**** password bit

ip subnet-zero

no ip domain-lookup

!

!

!

!

interface Loopback0

ip address 84.84.84.84 255.255.255.255

!

interface FastEthernet0

ip address 52.72.31.1 255.255.128.0

speed auto

half-duplex

no cdp enable

!

interface Serial0

no ip address

ip helper-address 52.200.127.255

ip directed-broadcast

encapsulation frame-relay

no fair-queue

frame-relay lmi-type ansi

!

interface Serial0.16 point-to-point

description Primary PVC to HQ

bandwidth 64

ip address 52.5.84.1 255.255.255.128

ip directed-broadcast

ip ospf authentication-key cisco

no cdp enable

frame-relay class data

frame-relay interface-dlci 16

!

interface Serial0.17 point-to-point

description Backup link to Long

bandwidth 64

ip address 52.5.84.129 255.255.255.128

ip directed-broadcast

ip ospf authentication-key cisco

ip ospf cost 4000

no cdp enable

frame-relay class data

frame-relay interface-dlci 17

!

router ospf 100

log-adjacency-changes

area 0.0.0.84 authentication

area 0.0.0.84 stub

area 0.0.0.100 authentication

area 0.0.0.100 nssa

redistribute connected metric 2000 metric-type 1 subnets

network 52.5.84.1 0.0.0.0 area 0.0.0.84

network 52.5.84.129 0.0.0.0 area 0.0.0.100

network 52.72.31.1 0.0.0.0 area 0.0.0.84

!

ip classless

ip forward-protocol udp 1604

ip route 0.0.0.0 0.0.0.0 52.5.84.130

no ip http server

!

!

!

map-class frame-relay data

no frame-relay adaptive-shaping

frame-relay cir 48000

frame-relay bc 48000

frame-relay be 16000

frame-relay mincir 48000

no cdp run

snmp-server community public RO

snmp-server community private RW

snmp-server host 52.200.91.65 private

!

**** password bit ****

!

end

***************

hq core router config

***************

!

version 12.2

service timestamps debug uptime

service timestamps log uptime

no service password-encryption

!

hostname hq

!

****************

loads of stuff about isdn left out

****************

memory-size iomem 10

ip subnet-zero

!

!

no ip domain-lookup

!

!

interface Loopback0

ip address 1.1.1.1 255.255.255.255

!

interface FastEthernet0/0

description LAN PORT

ip address 52.200.33.7 255.255.128.0

ip helper-address 52.200.127.255

no ip redirects

ip directed-broadcast

ip ospf authentication-key cisco

duplex auto

speed auto

no cdp enable

standby 1 ip 52.200.33.9

standby 1 priority 110

standby 1 preempt

!

interface Serial1/0

description main frame relay bearer

no ip address

encapsulation frame-relay

no ip mroute-cache

no fair-queue

serial restart_delay 0

frame-relay lmi-type ansi

!

************************

approx 30 other sub interfaces left out

************************

!

interface Serial1/0.134 point-to-point

description Primary PVC to Bar

bandwidth 64

ip address 52.5.84.2 255.255.255.128

ip helper-address 52.200.127.255

ip directed-broadcast

ip ospf authentication-key cisco

no ip mroute-cache

no cdp enable

frame-relay class data

frame-relay interface-dlci 34

frame-relay payload-compression packet-by-packet

interface Serial1/1

no ip address

shutdown

serial restart_delay 0

no cdp enable

!

interface Serial1/2

no ip address

shutdown

serial restart_delay 0

no cdp enable

!

interface Serial1/3

no ip address

shutdown

serial restart_delay 0

no cdp enable

*************************

load of dialer definitions left out

*************************

!

!

router ospf 100

log-adjacency-changes

area 0.0.0.0 authentication

area 0.0.0.84 authentication

area 0.0.0.84 stub

area 0.0.0.100 authentication

area 0.0.0.100 nssa default-information-originate

network 52.5.84.2 0.0.0.0 area 0.0.0.84

network 52.5.84.128 0.0.0.127 area 0.0.0.100

network 52.200.33.7 0.0.0.0 area 0.0.0.0

!

ip classless

ip forward-protocol udp 1604

ip route 0.0.0.0 0.0.0.0 52.200.33.1

no ip http server

ip pim bidir-enable

!

!

!

map-class frame-relay data

no frame-relay adaptive-shaping

frame-relay cir 48000

frame-relay bc 48000

frame-relay be 16000

frame-relay mincir 48000

dialer-list 1 protocol ip permit

no cdp run

snmp-server engineID local 00000009020000049A37F6A0

snmp-server community public RO

snmp-server community private RW

snmp-server host 52.200.91.65 private

!

***********

password bit

***********

!

!

end

*************

2nd core router config

*************

!

version 12.2

service timestamps debug uptime

service timestamps log uptime

no service password-encryption

!

hostname Long

!

***********

password bit

***********

!

memory-size iomem 10

ip subnet-zero

!

!

!

!

!

!

interface Loopback0

ip address 2.2.2.2 255.255.255.255

!

interface FastEthernet0/0

ip address 52.200.33.8 255.255.128.0

ip helper-address 52.200.127.255

no ip redirects

ip directed-broadcast

ip ospf authentication-key cisco

duplex auto

speed auto

no cdp enable

standby 1 ip 52.200.33.9

standby 1 preempt

!

interface Serial1/0

no ip address

encapsulation frame-relay

no ip mroute-cache

no fair-queue

serial restart_delay 0

frame-relay lmi-type ansi

!

***************

sub interfaces missed out

***************

!

interface Serial1/0.148 point-to-point

description Primary PVC to Bar

bandwidth 64

ip address 52.5.84.130 255.255.255.128

ip helper-address 52.200.127.255

ip directed-broadcast

ip ospf authentication-key cisco

ip ospf cost 4000

no ip mroute-cache

no cdp enable

frame-relay class data

frame-relay interface-dlci 48

frame-relay payload-compression packet-by-packet

interface Serial1/1

no ip address

shutdown

serial restart_delay 0

no cdp enable

!

interface Serial1/2

no ip address

shutdown

serial restart_delay 0

no cdp enable

!

interface Serial1/3

no ip address

shutdown

serial restart_delay 0

no cdp enable

!

router ospf 100

log-adjacency-changes

area 0.0.0.0 authentication

area 0.0.0.84 authentication

area 0.0.0.84 stub

area 0.0.0.100 authentication

area 0.0.0.100 nssa default-information-originate

network 52.5.84.0 0.0.0.127 area 0.0.0.84

network 52.5.84.130 0.0.0.0 area 0.0.0.100

network 52.200.33.8 0.0.0.0 area 0.0.0.0

!

ip classless

ip forward-protocol udp 1604

ip route 0.0.0.0 0.0.0.0 52.200.33.1

no ip http server

ip pim bidir-enable

!

!

!

map-class frame-relay data

no frame-relay adaptive-shaping

frame-relay cir 48000

frame-relay bc 48000

frame-relay be 16000

frame-relay mincir 48000

!

map-class frame-relay 3com

no frame-relay adaptive-shaping

frame-relay cir 64000

frame-relay bc 64000

frame-relay be 0

frame-relay mincir 64000

no cdp run

snmp-server engineID local 00000009020000049A134B90

snmp-server community private RW

snmp-server community public RO

snmp-server host 52.200.91.65 private

!

***************************

password bit

***************************

!

end

\f1

}

[tomredmond]

re my message above I forgot to send output to terminal DOH! here is the result of the DEBUG IP OSPF ADJ

19:17:32: OSPF: Send DBD to 2.2.2.2 on Serial0.17 seq 0x2CC opt 0x48 flag 0x7 len 32

19:17:32: OSPF: Retransmitting DBD to 2.2.2.2 on Serial0.17 [1]

19:17:33: OSPF: Rcv hello from 1.1.1.1 area 0.0.0.84 from Serial0.16 52.5.84.2

19:17:33: OSPF: End of hello processing

19:17:33: OSPF: Send DBD to 1.1.1.1 on Serial0.16 seq 0x1954 opt 0x40 flag 0x7 len 32

19:17:33: OSPF: Retransmitting DBD to 1.1.1.1 on Serial0.16 [4]

19:17:37: OSPF: Send DBD to 2.2.2.2 on Serial0.17 seq 0x2CC opt 0x48 flag 0x7 len 32

19:17:37: OSPF: Retransmitting DBD to 2.2.2.2 on Serial0.17 [2]

19:17:37: OSPF: Rcv hello from 2.2.2.2 area 0.0.0.100 from Serial0.17 52.5.84.130

19:17:37: OSPF: End of hello processing

********

many repeats of above

********

19:18:23: OSPF: Killing nbr 1.1.1.1 on Serial0.16 due to excessive (13) retransmissions

19:18:23: OSPF: 1.1.1.1 address 52.5.84.2 on Serial0.16 is dead, state DOWN

19:18:23: %OSPF-5-ADJCHG: Process 100, Nbr 1.1.1.1 on Serial0.16 from EXSTART to DOWN, Neighbor Down

: Too many retransmitio

19:18:27: OSPF: Send DBD to 2.2.2.2 on Serial0.17 seq 0x2CC opt 0x48 flag 0x7 len 32

19:18:27: OSPF: Retransmitting DBD to 2.2.2.2 on Serial0.17 [12]

19:18:27: OSPF: Rcv hello from 2.2.2.2 area 0.0.0.100 from Serial0.17 52.5.84.130

19:18:27: OSPF: End of hello processing

19:18:27: OSPF: Build router LSA for area 0.0.0.84, router ID 84.84.84.84, seq 0x8000001D

19:18:27: OSPF: Build router LSA for area 0.0.0.100, router ID 84.84.84.84, seq 0x8000001D

19:18:32: OSPF: Send DBD to 2.2.2.2 on Serial0.17 seq 0x2CC opt 0x48 flag 0x7 len 32

19:18:32: OSPF: Retransmitting DBD to 2.2.2.2 on Serial0.17 [13]

19:18:33: OSPF: Rcv hello from 1.1.1.1 area 0.0.0.84 from Serial0.16 52.5.84.2

19:18:33: OSPF:

OSPF: Nbr 1.1.1.1 52.5.84.2 Serial0.16 is currently ignored

19:18:37: OSPF: Killing nbr 2.2.2.2 on Serial0.17 due to excessive (13) retransmissions

19:18:37: OSPF: 2.2.2.2 address 52.5.84.130 on Serial0.17 is dead, state DOWN

19:18:37: %OSPF-5-ADJCHG: Process 100, Nbr 2.2.2.2 on Serial0.17 from EXSTART to DOWN, Neighbor Down

: Too many retransmitions

19:18:37: OSPF: Rcv hello from 2.2.2.2 area 0.0.0.100 from Serial0.17 52.5.84.130

19:18:37: OSPF:

OSPF: Nbr 2.2.2.2 52.5.84.130 Serial0.17 is currently ignored

bar#

19:18:43: OSPF: Rcv hello from 1.1.1.1 area 0.0.0.84 from Serial0.16 52.5.84.2

19:18:43: OSPF:

OSPF: Nbr 1.1.1.1 52.5.84.2 Serial0.16 is currently ignored

19:18:47: OSPF: Rcv hello from 2.2.2.2 area 0.0.0.100 from Serial0.17 52.5.84.130

19:18:47: OSPF:

OSPF: Nbr 2.2.2.2 52.5.84.130 Serial0.17 is currently ignored

bar#no debug all

All possible debugging has been turned off

bar#

[mcmarcos]

Can you provide us the routing table?

I think you cannot do this at the same time:

redistribute connected metric 2000 metric-type 1 subnets

network 52.72.31.1 0.0.0.0 area 0.0.0.84

Since the 52.72.31.1 is the addr of eth and you put this net on the area 84, the redistributed connected do this for you.

And i think too that area 84 should be nssa.

[deilert]

Tom

Prior to looking at the OSPF at L3 I would ensure my PVC's were mapped properly.

My suggestions

1. turn on cdp see what you get. make sure your DLCI's at the remote side are mapped to the proper interfaces, the carrier could just have them reveresed.

2. if every thing is mappered properly , remove the FR traffic shaping parameters.

It looks to me that your PVC's are not mapped properly

[tomredmond]

I did what you asked, the result is below. The crossed pvc's did occur to me and I checked with the provider and I think the results below show that they are ok.

However one odd thing on debug ip ospf adj output I noticed a line stating "could not see ourselves in hello" . Hello packets are being traded but the negotiation seems to be failing.

There is no frame-relay adaptive shaping.

Bar#sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

S - Switch, H - Host, I - IGMP, r - Repeater

Device ID Local Intrfce Holdtme Capability Platform Port ID

hq Ser 0.16 134 R 3660 Ser 1/0.134

Long Ser 0.17 133 R 3660 Ser 1/0.148