- Config change made to replace the MAC address in an IP DHCP Pool
- After change the Serial Interface started to flap continuously (thus BGP adjacency changes also)
- LAN cable removed and the flapping stops (after approx 5mins)
- The 2821 has been fully swapped out
- LAN cables replaced
- Swapping LAN switch tomorrow
Anyone have any clues where I should investigate next?
TAC guys bamboozled with this today..... help...
IP/Analogue Gateway <-> Lan Switch (c3560) <-> CE (c2821) <-> PE (7206)
Many thanks in advance.
A most unusual experience indeed. Please allow me a couple of questions:
1.) When the flapping occurs, what exactly is flapping - only the line protocol on the Serial interface, or also the physical state? Also, does the flapping have a regular period?
2.) During the flapping, is the CPU or interrupt load significantly increased?
3.) Can you check if there is no broadcast or unicast storm in the switched network?
4.) Does the flapping also happen when you connect the LAN interface to a silent machine, e.g. a notebook that generates only little or no traffic?
5.) What link protocol is used on the Serial link - Frame Relay, PPP, HDLC, ...? What about Layer3 - only IPv4 or also IPv6? Also, is your configuration in any way unusual (e.g., bridging instead of switching)? Is it possible to post the configuration of your 2821 here?
6.) Are there any interesting figures displayed in the "show interfaces serial" counters? Can this output be also posted here?
7.) The 2821 has two LAN ports if I'm not mistaken. Does this problem also appear when you swap the LAN interfaces (also moving the configuration from one LAN interface to another)?
Sorry for asking so much - I'm trying to get a clue of what might be happening.
Thanks for your interest. I am currently out of the office so can only answer some of the questions posed (for now).
3. Since the only (remote) connection is via the WAN serial interface I am unable to access the LAN Switch. The flapping prevents access. I am looking at a workaround for this.
4. Yes. I have connected a laptop to the Gig0/0 interface and the flapping occured.
5. WAN Serial - frame-relay. Performed debug Frame-relay PVC 101 & debug fram-relay map earlier with TAC Eng. Checked out OK. IPv4 only.
6. SH INT SERIAL - 0 CRC's; numerous (100's) carrier transitions - counting up when the LAN connection is up.
7. Yes. Tried the LAN cable in Gig0/1. Flapping occured after a couple of minutes of connection being up/up.
Please feel free to ask more questions. I would be delighted to determine the Root Cause.
Thank you very much for providing these answers. I'm looking forward to seeing the answers to my questions 1-3 (you've answered the 3 briefly but we have to be sure about the broadcast/multicast storm).
For today, only one more question: what exactly was the change that started this behavior, and does the flappning stop when that change is undone?
Trigger of this event:
- The MAC address was changed in an IP DHCP POOL.
- I copied the IP DHCP Pool to notepad
- Changed to the new MAC Address as swapping the IP/Analogue Gateway
- Conf t & pasted it in
- 2 mins later the flapping occured
Sep 3 08:07:58.407 BST: %SYS-5-CONFIG_I: Configured from console by jmc on vty0 (220.127.116.11)
Sep 3 08:10:10.883 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
Sep 3 08:10:13.315 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
Sep 3 08:10:49.319 BST: %BGP-5-ADJCHANGE: neighbor x.x.x.x Down BGP Notification sent
Sep 3 08:10:49.319 BST: %BGP-3-NOTIFICATION: sent to neighbor x.x.x.x 4/0 (hold time expired) 0 bytes
Sep 3 08:11:20.711 BST: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up
Sep 3 08:11:52.386 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
Sep 3 08:11:53.386 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to down
Sep 3 08:11:55.458 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
Sep 3 08:11:56.458 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to up
Sep 3 08:12:03.906 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
Sep 3 08:12:06.210 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
Sep 3 08:12:35.906 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
Sep 3 08:12:38.082 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
Sep 3 08:12:41.082 BST: %BGP-5-ADJCHANGE: neighbor x.x.x.x Down BGP Notification sent
Sep 3 08:12:41.082 BST: %BGP-3-NOTIFICATION: sent to neighbor x.x.x.x 4/0 (hold time expired) 0 bytes
Sep 3 08:13:19.410 BST: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up
Sep 3 08:13:55.010 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
Sep 3 08:13:57.314 BST: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
- I have undone the change and the flapping still remains when the LAN cable is connected
With regards to the previous questions 1 & 2.... It is very difficult to get onto the box long enough for the commands to be ran. I will see if I can get the info with the LAN disconnected.
Many thanks for your help so far
Thanks a lot for your reply.
I know this is a bit intrusive but is it possible to post the entire configuration of the 2821 here? (Sensitive information blanked out, of course)
Busy, busy. This is the current status..
- After all equipment swapped (LAN,CE, NTE, PE Port, all cables) flapping remained.
- Troubleshooting continued by making changes to the CE Config.
- Keepalives were removed leaving the LAN interface up/up (after LAN cable removed)
- The 4 Voice Vlans were advertising via BGP utilising a contiguous subnet
- These were split into 4 separate vlans and advertised out separately
- The Voice Vlan 1 was added back in and flapping started
- Voice Vlan 1 was removed flapping stopped
- Voice Vlan 2 added - NO flapping
- Voice Vlan 3 & 4 added - NO flapping
So it appears that as soon as this subnet was advertised out to the WAN the flapping occured.
The network has been scanned and no duplicate subnet exists.
Although we have a workaround the Mystery continues.......
Thanks for any advice offered!!
Is it perhaps possible that by advertising the IP network for VLAN 1, you have advertised a destination network into which a huge data traffic started to flow from the outside? Purely theoretically, that could interfere with the keepalive mechanism of your serial interface.
Please let me ask once again: is it possible for you to post here the configuration of the CE router that ecountered the flapping problem?