cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1226
Views
0
Helpful
6
Replies

Help with 7206VXR

Ken Hagen
Level 1
Level 1

We have 2cisco 7206VXR (NPE300) processor (revision D) with 229376K/65536K bytes of memory. connected by multilink. We've been losing connetion over the multilink every night.  It was ok for several years until recently. Anyone had any issues like this before?  To make it come back up, we can go to the local 7204 and telnet to the remote over a ds0 that is not in the multilink.  We configured one ds0 out of the ds1 for the purpose of bringing the link back up.  The multilink goes down and we can't get it back up otherwise.  There is a ds1 in multilink .  The ios is c7200-jk2s-mz.121-27b.E3.bin.  The memory is 

 

show memory
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 626FBC20 194003936 19941772 174062164 169627356 169475272
I/O 20000000 33554432 615904 32938528 32786424 32660828
I/O-2 E000000 33554512 3240880 30313632 30313632 30310716

 

 

 

The proc cpu is:

sh proc cpu
CPU utilization for five seconds: 15%/11%; one minute: 19%; five minutes: 21%

 

 

 

multilink is:

sh int multilink 1
Multilink1 is up, line protocol is up
Hardware is multilink group interface
Internet address is x.x.x.x/30
MTU 1500 bytes, BW 35328 Kbit, DLY 100000 usec,
reliability 255/255, txload 3/255, rxload 11/255
Encapsulation PPP, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 2 seconds on reset
LCP Open, multilink Open
Open: IPCP
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 3d00h
Input queue: 41/75/55145/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1647000 bits/sec, 279 packets/sec
5 minute output rate 458000 bits/sec, 246 packets/sec
113102407 packets input, 3977303185 bytes, 50 no buffer
Received 0 broadcasts (0 IP multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
101205152 packets output, 258441157 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions

Any thoughts on what is causing this? Anyone had this issue?  Thanks for any suggestions.

 

 

 

KEN HAGEN CCIE#56460
SENIOR NETWORK ENGINEER | PSA
SEATTLE CITY LIGHT
M: 206-459-0027 | ken.hagen@seattle.gov
We Power Seattle seattle.gov/city-light

6 Replies 6

chrihussey
VIP Alumni
VIP Alumni

Have some questions:

1- Could you please provide details on how the multilink is configured on either side? It looks almost like you are bundling 23 or more T1s. Is that the case?

2- What exactly are you doing to bring the multilink back up? (ie shut, no shut on the multilink interface)

3- What do the controllers look like for the PPP interfaces?

4- Can you post the output of "sh ppp multilink"?

5- Can you post the configs for the routers on either side?

6- Is there anying in the logs to indicate what may be happening?

 

Sorry for all the questions, there may be more, just trying to help.

 

 

 

 

 

1,

Local_router#sh run int multilink 1
Building configuration...

Current configuration : 153 bytes
!
interface Multilink1
ip address x.x.x.x 255.255.255.252
ppp multilink
ppp multilink group 1
ppp multilink fragment disable
no cdp enable
end
remote_router#sh run int multilink 1
Building configuration...

Current configuration : 242 bytes
!
interface Multilink1
ip address x.x.x.x 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
no ip route-cache
no ip mroute-cache
no cdp enable
ppp multilink
no ppp multilink fragmentation
multilink-group 1
end


yes, 23 channels are in the multilink and one is not.

2. I go to the local router and ssh to the remote router from the local router. This brings back up the connection.

3. Here is one of the ppp interfaces controllers:

sh controller s1/0/6:0
Serial1/0/6:0
Hardware versions:
firmware version: 0x01090000
iofpga version: 0x00020001
freedm version: 0x00000000
temux version: 0x00020003
subrate fpga version: 0x00010004
liu version: 0x00000000
Rommon version: 0x00020000
rx_int_count 320922135, tx_int_count 109041726
pak_move_failed 0 rx_idb_null_count 0
tx_count 0, output_qcount 0
xoff_word 0x38377804, actual word 0x55757557,
bitword_mask_hi 0xC0 ,bitword_mask_lo 0x30
rtn_code_hi 0x0, rv_hi 0x40
rtn_code_lo 0x0, rv_lo 0x10
HIQ XON R=0 V=1
LOQ XON R=0 V=1
hiq is XON hiq_seen 2
loq is XON
bytes_in_tx_lo 194, bytes_threshold_lo 1920
bytes_in_tx_hi 66, bytes_threshold_hi 1920
unmatched_anyphy 0 noshim 0

interface Serial1/0/6:0 is not master_link

fs_safe_start_fail 6151015 fs_safe_start_pass 6512968
Max Output enqueued 1, queue_drops 0
safe_str_tail 78, safe_str_head 78
fastsend is 0x347124, safestart is 0x348218
oqueue 0x2793C8, oqueue_dequeue 0x2F026F8
write_addr 0xE0710000
rx_dma_bd_index_addr 0x3837A040, rxr 0xE0704000, rxr_shadow 0x96EB118
present valid buffer pointer is 0xC03A4A38, in shadow 0x384A3AC0
Tx interrupt 0x3482E4
ci_table 0x9534664
ctx 0x9533348
oir_on_wait:FALSE no_intfs_reset:0x00000000
Autodnr Lock fail 0 Autodnr Process Failures 0
shadow_tx_status 0x38377804 Active VCs/Port I port 24 II port 0
IPC stats for slot 1
0x94E5A38: ipc_rx_pkt_drops 0, ipc_tx_pkt_drops 0
pa_ipc_restart_cnt 0, pa_ipc_keep_restart 0

No Keepalive 0 ECC Single bit errors 0 tx_fifo_avail_threshold 0x38377800 value 0

4.
Local_router#sh ppp multi

Multilink1
Bundle name:
Remote Endpoint Discriminator: [1] Remote
Local Endpoint Discriminator: [1] Local
Bundle up for 11:37:43, total bandwidth 35328, load 39/255
Receive buffer limit 276000 bytes, frag timeout 1000 ms
0/0 fragments/bytes in reassembly list
0 lost fragments, 2355064 reordered
0/0 discarded fragments/bytes, 0 lost received
0xD365B0 received sequence, 0x17C268 sent sequence
Member links: 23 active, 0 inactive (max 255, min not set)
Se1/0/28:0, since 11:37:43
Se1/0/27:0, since 11:37:43
Se1/0/26:0, since 11:37:43
Se1/0/25:0, since 11:37:43
Se1/0/24:0, since 11:37:43
Se1/0/23:0, since 11:37:43
Se1/0/22:0, since 11:37:43
Se1/0/21:0, since 11:37:43
Se1/0/20:0, since 11:37:43
Se1/0/19:0, since 11:37:43
Se1/0/18:0, since 11:37:43
Se1/0/16:0, since 11:37:43
Se1/0/17:0, since 11:37:43
Se1/0/15:0, since 11:37:43
Se1/0/14:0, since 11:37:43
Se1/0/13:0, since 11:37:43
Se1/0/12:0, since 11:37:43
Se1/0/11:0, since 11:37:43
Se1/0/10:0, since 11:37:43
Se1/0/9:0, since 11:37:43
Se1/0/8:0, since 11:37:43
Se1/0/7:0, since 11:37:43
Se1/0/6:0, since 11:37:43
No inactive multilink interfaces
5.

version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname local
!
boot-start-marker
boot system flash disk2:c7200p-adventerprisek9-mz.152-4.M11.bin
boot bootldr bootflash:c7200p-kboot-mz.152-4.M11.bin
boot-end-marker
!
!
logging exception 32762
no logging console
enable password 7 xxxxxx
!
aaa new-model
!
!
!
!
!
!
!
aaa session-id common
clock calendar-valid
!
!
!
!
!
!
no ip domain lookup
ip domain name x.x.x.x
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
username x password 7 y
!
redundancy
!
!
controller T3 1/0
clock source line
cablelength 45
error-throttling
t1 5 channel-group 0 timeslots 1-24
t1 6 channel-group 0 timeslots 1-24
t1 7 channel-group 0 timeslots 1-24
t1 8 channel-group 0 timeslots 1-24
t1 9 channel-group 0 timeslots 1-24
t1 10 channel-group 0 timeslots 1-24
t1 11 channel-group 0 timeslots 1-24
t1 12 channel-group 0 timeslots 1-24
t1 13 channel-group 0 timeslots 1-24
t1 14 channel-group 0 timeslots 1-24
t1 15 channel-group 0 timeslots 1-24
t1 16 channel-group 0 timeslots 1-24
t1 17 channel-group 0 timeslots 1-24
t1 18 channel-group 0 timeslots 1-24
t1 19 channel-group 0 timeslots 1-24
t1 20 channel-group 0 timeslots 1-24
t1 21 channel-group 0 timeslots 1-24
t1 22 channel-group 0 timeslots 1-24
t1 23 channel-group 0 timeslots 1-24
t1 24 channel-group 0 timeslots 1-24
t1 25 channel-group 0 timeslots 1-24
t1 26 channel-group 0 timeslots 1-24
t1 27 channel-group 0 timeslots 1-24
t1 28 channel-group 0 timeslots 1-24
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
ip address x.x.x.x 255.255.255.255
!
interface Multilink1
ip address x.x.x.x 255.255.255.252
ppp multilink
ppp multilink group 1
ppp multilink fragment disable
no cdp enable
!
interface GigabitEthernet0/1
ip address x.x.x.x 255.255.255.254
duplex auto
speed auto
media-type rj45
negotiation auto
no cdp enable
!
interface FastEthernet0/2
ip address dhcp
duplex auto
speed auto
!
interface GigabitEthernet0/2
ip address x.x.x.x 255.255.255.254
duplex auto
spee media-type rj45
negotiation auto
no cdp enable
!
interface GigabitEthernet0/3
no ip address
shutdown
duplex auto
speed auto
media-type rj45
negotiation auto
!
interface Serial1/0/5:0
ip address x.x.x.x 255.255.255.254
no ip proxy-arp
no keepalive
no cdp enable
!
interface Serial1/0/6:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/7:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/8:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/9:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/10:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/11:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable d auto
interface Serial1/0/12:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/13:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/14:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/15:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/16:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/17:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/18:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
interface Serial1/0/19:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/20:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/21:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/22:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/23:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/24:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/25:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
interface Serial1/0/26:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/27:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
interface Serial1/0/28:0
no ip address
encapsulation ppp
no keepalive
ppp multilink
ppp multilink group 1
no cdp enable
!
router ospf 1
router-id x.x.x.x
auto-cost reference-bandwidth 1000000
area 1 stub no-summary
passive-interface default
no passive-interface GigabitEthernet0/1
no passive-interface GigabitEthernet0/2
no passive-interface Multilink1
network x.x.x.x 0.0.0.3 area 1
network x.x.x.x 0.0.0.1 area 1
network x.x.x.x 0.0.0.1 area 1
network x.x.x.x 0.0.0.255 area 1
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip route
ip route
ip route
ip route
ip route
ip route
ip route
ip route
ip route
ip route
ip route
!
logging source-interface Loopback0
logging host x.x.x.x
access-list 20 permit x.x.x.x 0.0.255.255
access-list 20 permit x.x.x.x 0.0.255.255
access-list 20 permit x.x.x.x 0.0.255.255
access-list 20 deny any log
no cdp run
snmp-server community string RO
snmp-server contact Network Administration
snmp-server enable traps entity-sensor threshold
!
!
!
!
control-plane
!
!
!
mgcp profile default
!
!
!
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
access-class 20 in
exec-timeout 20 0
password 7 password
history size 200
transport input ssh
!
ntp source Loopback0
ntp update-calendar
ntp server x.x.x.x
ntp server x.x.x.x
ntp server x.x.x.x
ntp server x.x.x.x prefer
!
end


6. nothing in the logs
KEN HAGEN CCIE#56460
SENIOR NETWORK ENGINEER | PSA
SEATTLE CITY LIGHT
M: 206-459-0027 | ken.hagen@seattle.gov
We Power Seattle seattle.gov/city-light

It is odd that simply establishing an ssh connection brings the multilink back up. 

I can only make suggestions at this point:

1- Are the multilink interfaces up/down when this happens?

2- Do you see the input / output counters at the local site increment in any way?

3- Prior to ssh-ing to the remote, have you tried to just ping across the link to see if that wakes the remote up?

4 - While down, does a debug ppp data or events provide any insight?

5- Was the logging buffer empty? I would assume there would be something about the OSPF neighbors or interfaces going down. If not enabled then enable logging on both devices. "logging buffered informational"

6- Can you check the T3 controllers to make sure they are clean? (clear counters if necessary).

7- How long have the routers been up? (A reload may just do the trick).

That's all I have for now.

 

I will be able to look at this tonight. Thanks for the suggestions.
KEN HAGEN CCIE#56460
SENIOR NETWORK ENGINEER | PSA
SEATTLE CITY LIGHT
M: 206-459-0027 | ken.hagen@seattle.gov
We Power Seattle seattle.gov/city-light

1. The multilinks are up when it happens
2. Counters are incrementing
3. Pinging does bring it up too.
4. I didn't get a chance to do debug while it was down yet. Will try it tomorrow.
5. Logging is enabled and it shows the line protocol going down. Seems to point to service provider.
6. Controllers look clean
7. The routers have only been up one week. I believe a co-worker tried a reload.

KEN HAGEN CCIE#56460
SENIOR NETWORK ENGINEER | PSA
SEATTLE CITY LIGHT
M: 206-459-0027 | ken.hagen@seattle.gov
We Power Seattle seattle.gov/city-light

It could very well be a service provider issue.

In normal operation are OSPF neighbors established? And what is their state when it is down?

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco